Skip to content
This repository has been archived by the owner on Apr 21, 2023. It is now read-only.

large files and we loop forever #14

Closed
jeffkaufman opened this issue Oct 31, 2012 · 9 comments
Closed

large files and we loop forever #14

jeffkaufman opened this issue Oct 31, 2012 · 9 comments

Comments

@jeffkaufman
Copy link
Contributor

In jefftk-pagespeed-resource-handler, large resource (non-html) files get us into into an infinite loop:

2012/10/31 17:56:59 [debug] 1971#0: *4 write old buf t:0 f:0 00007F2830022B80, pos 00007F2830022B80, size: 0 file: 0, size: 0
2012/10/31 17:56:59 [debug] 1971#0: *4 write old buf t:0 f:0 0000000000000000, pos 000000000094637A, size: 5 file: 0, size: 0
2012/10/31 17:56:59 [debug] 1971#0: *4 write old buf t:0 f:0 00007F2830022B80, pos 00007F2830022B80, size: 0 file: 0, size: 0
2012/10/31 17:56:59 [debug] 1971#0: *4 write old buf t:0 f:0 0000000000000000, pos 000000000094637A, size: 5 file: 0, size: 0
...

This consistently happens for me after producing 49019 bytes of output. It happens with both javascript files and images.

In jefftk-async-rewrite I can get the same thing if I disable the "only rewrite html resources" check (at the beginning of ngx_http_pagespeed_body_filter).
@bnoordhuis
Copy link
Contributor

What do you see when you attach strace to the worker?

@yaoweibin
Copy link
Contributor

How to test with this problem? I checkout your code, and it seems work for me in my box. And I saw the image are from the static file handler.

In Nginx, there is no larger buffer than 32k, can you split your big buffer to small chunk with a chain? It could save the memory. And all the small chunk buffers can be recycled.

@jeffkaufman
Copy link
Contributor Author

@yaoweibin

How to test with this problem?

Try loading "/Puzzle.jpg.pagespeed.ce.91_WewrLtP.jpg" on jefftk-pagespeed-resource-handler after setting up nginx to load from https://github.com/pagespeed/ngx_pagespeed/tree/jefftk-pagespeed-resource-handler/test/www .

there is no larger buffer than 32k

That's probably the problem. When we pull from pagespeed we try to allocate an nginx buffer big enough for whatever is in the pagespeed buffer right now:

b->start = b->pos = static_cast<u_char*>(
  ngx_palloc(request_->pool, buffer_.length()));

I'll change that to split into multiple nginx buffers if the pagespeed buffer is big.

@jeffkaufman
Copy link
Contributor Author

I've added buffer-splitting to jefftk-resource-handler [1]. it hasn't fixed the problem. Still working on it.

[1] which has replaced jefftk-pagespeed-resource-handler and is based off master instead of jefftk-async-rewrite

@yaoweibin
Copy link
Contributor

On 2012-11-1 23:04, Jeff Kaufman wrote:

I've added buffer-splitting to jefftk-resource-handler [1]. it hasn't
fixed the problem. Still working on it.

[1] which has replaced jefftk-pagespeed-resource-handler and is based
off master instead of jefftk-async-rewrite

—

I'll check this problem tomorrow.

Thanks,
-Weibin Yao

@jeffkaufman
Copy link
Contributor Author

It looks like something is keeping buffers from going out, and before it would build up a lot of empty buffers. In fefdac7 I made it stop producing empty buffers, so now it's just stuck on real content-full buffers.

@jeffkaufman
Copy link
Contributor Author

2012/11/01 17:55:44 [debug] 28441#0: *1 http write filter: l:1 f:0 s:192201
2012/11/01 17:55:44 [debug] 28441#0: *1 http write filter limit 0
2012/11/01 17:55:44 [debug] 28441#0: *1 http write filter 00000000027235F0
2012/11/01 17:55:44 [debug] 28441#0: timer delta: 0
2012/11/01 17:55:44 [debug] 28441#0: posted events 0000000000000000
2012/11/01 17:55:44 [debug] 28441#0: worker cycle
2012/11/01 17:55:44 [debug] 28441#0: epoll timer: -1
2012/11/01 17:55:44 [debug] 28441#0: epoll: fd:9 ev:0010 d:00007F08F2026238
2012/11/01 17:55:44 [debug] 28441#0: epoll_wait() error on fd:9 ev:0010
Would lock buffer_
Would unlock buffer_
2012/11/01 17:55:44 [debug] 28441#0: *1 http postpone filter "/Puzzle.jpg.pagespeed.ce.91_WewrLtP.jpg?" 0000000000000000
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002714D70, pos 0000000002716D13, size: 93 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002716D80, pos 0000000002716D80, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002718D90, pos 0000000002718D90, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000271ADA0, pos 000000000271ADA0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000271CDB0, pos 000000000271CDB0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000271EDC0, pos 000000000271EDC0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002720DD0, pos 0000000002720DD0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002723DF0, pos 0000000002723DF0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002725E00, pos 0000000002725E00, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002727E10, pos 0000000002727E10, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002729E20, pos 0000000002729E20, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000272BE30, pos 000000000272BE30, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000272DE40, pos 000000000272DE40, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000272FE50, pos 000000000272FE50, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002731E60, pos 0000000002731E60, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002733E70, pos 0000000002733E70, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002735E80, pos 0000000002735E80, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002737E90, pos 0000000002737E90, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002739EA0, pos 0000000002739EA0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000273BEB0, pos 000000000273BEB0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000273DEC0, pos 000000000273DEC0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 000000000273FED0, pos 000000000273FED0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002741EE0, pos 0000000002741EE0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002743EF0, pos 0000000002743EF0, size: 8192 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002745F20, pos 0000000002745F20, size: 3692 file: 0, size: 0
2012/11/01 17:55:44 [debug] 28441#0: *1 http write filter: l:1 f:0 s:192201
2012/11/01 17:55:44 [debug] 28441#0: *1 http write filter limit 0
2012/11/01 17:55:44 [debug] 28441#0: *1 http write filter 00000000027235F0
2012/11/01 17:55:44 [debug] 28441#0: timer delta: 1
2012/11/01 17:55:44 [debug] 28441#0: posted events 0000000000000000
2012/11/01 17:55:44 [debug] 28441#0: worker cycle
2012/11/01 17:55:44 [debug] 28441#0: epoll timer: -1
2012/11/01 17:55:44 [debug] 28441#0: epoll: fd:9 ev:0010 d:00007F08F2026238
2012/11/01 17:55:44 [debug] 28441#0: epoll_wait() error on fd:9 ev:0010
Would lock buffer_
Would unlock buffer_
2012/11/01 17:55:44 [debug] 28441#0: *1 http postpone filter "/Puzzle.jpg.pagespeed.ce.91_WewrLtP.jpg?" 0000000000000000
2012/11/01 17:55:44 [debug] 28441#0: *1 write old buf t:1 f:0 0000000002714D70, pos 0000000002716D13, size: 93 file: 0, size: 0
...

@jeffkaufman
Copy link
Contributor Author

http://www.jefftk.com/nginx-misc/2012-11-01--puzzle-crash.log.gz

@jeffkaufman
Copy link
Contributor Author

Fixed by @yaoweibin in pull request #18

@yaoweibin yaoweibin mentioned this issue Nov 10, 2012
Closed
@oschaaf oschaaf mentioned this issue Nov 27, 2012
Closed
@e1ven e1ven mentioned this issue Apr 26, 2013
Open
@melong007 melong007 mentioned this issue Apr 22, 2014
Closed
@oschaaf oschaaf mentioned this issue Aug 8, 2014
Closed
@morlovich morlovich mentioned this issue Dec 9, 2015
Closed
@Hujinyong Hujinyong mentioned this issue Jan 12, 2017
Open
@morlovich morlovich mentioned this issue May 22, 2018
Open
@lanki567 lanki567 mentioned this issue Oct 3, 2019
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yaoweibin @jeffkaufman @bnoordhuis