Crash on very long urls #1248
Comments
|
What version/os/64 or 32 bits?
|
|
Apache version: 2.2 Thanks! |
|
Also what version of mod_pagespeed?
|
|
ModSpeed version: Version: 1.9.32.11 |
|
I saw similar issue reported #334 (looks pretty old ticket) - and was mentioned that it was resolved in Release 0.10.22.4 (https://developers.google.com/speed/pagespeed/module/release_notes#release_1.9.32.11-stable) |
|
To add a few more information:
|
|
A few more logs details we pulled from apache log: Tue Jan 19 04:30:28 2016] [error] [mod_pagespeed 1.9.32.11-7550 @9909] ..... |
|
great, thanks. I'll symbolize the backtrace tomorrow morning and hopefully
|
|
A few more info: We noticed that somehow this error is related to the crash (most of the time): |
|
if you still have the process in gdb, could you do |
|
Unfortunately it says proc folder not found However out team gave some more insight from the coredump and below is details |
|
Can you do info sharedlibrary ?
|
|
Please find the attached image - I could not format is well here |
|
Some more data!
URI too long -> No of time we see this error log We could not find yet why Apache is reporting long URL as we don't see that in our apache access log yet! |
|
is the backtrace from the same process as the |
|
Can you glance at the access.log? This message does come from -Josh On Wed, Jan 20, 2016 at 10:39 AM, Jeffrey Crowell notifications@github.com
|
|
@jmarantz I ran splunk query from those IP address and I could not find any URI that is long during that time window or from any other request at all. Is it possible that URI was not too long but it was something else reported as URI too long? @crowell - Yes both were run against the same coredump! Strange! |
|
not doubting the relation between the crashes and the message, but that http://apache2.sourcearchive.com/documentation/2.2.16-4/protocol_8c-source.html On Wed, Jan 20, 2016 at 1:57 PM, Joshua Marantz notifications@github.com
Jeff Crowell |
|
I am wondering since process crashed, may explain why I am not finding this 'long URI' on access log. Does that make sense? |
|
I'm also suspicious that the crash is caused by an out-of-memory condition. I'm wondering if there are any other messages in the error log that might shed light on the cause? Also, you mentioned above you have three filters enabled: rewrite_javascript, rewrite_css, and collapse_whitespace. Can you try turning them all off and seeing if the problem shows up? If it doesn't show up, can you add them one at a time, and see which one causes the problem? Thanks! |
|
I am not sure if it is related to memory. So far we have not seen memory issue reported but I will dig in to those area. Every time crashing with URI too long make me think it may not be related to memory crash but I could be complete wrong. We were going to recreate this issue on non-prod environment to simulate these scenarios as well. Without getting in to details initial thought apache was returning empty response with 414 status code and pagespeed module crashed with null exception
|
|
You are probably right. If we can get an accurate symbolized stack trace On Wed, Jan 20, 2016 at 7:58 PM, praveen262k notifications@github.com
|
|
The I'm going to try to reproduce this here, triggering |
|
To hit the request too large error message, the path is: This looks pretty normal, and like I should be able to repro by sending an 8k+ request line. |
|
Reproduced! |
|
The problem is that |
|
symbolized from the other bt, so yes it is the same issue 👍 (note to self |
|
Turns out we're checking for this exact problem, just too late: |
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
|
@crowell @jeffkaufman @jmarantz - This is great and was quick turnaround and really appreciate the attention and prompt response! Thanks |
|
Thanks! We're going to prepare an updated release that has this fix, and we're hoping to get that out early next week. |
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
apply befa494 Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
jeffkaufman
changed the title
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
|
Yet not successful finding the long URI - not in apache, F5, DynaTrace, CDN! CDN came back with list of URL who total bytes exceed 8190 - I tried to dig apache code and I could not understand if the header or other attributes were included along with URI to come up with that number (8190 as configured in Apache) and throw error. http://code.metager.de/source/xref/apache/httpd/server/protocol.c |
|
@praveen262k yes, the 8190 is a limitation of apache. with the patch in mod_pagespeed, your server shouldn't crash any more, but we can't work around the limitation in apache. |
|
@crowell Thanks! I was trying to find the actual URL because we don't expect anything of that large URI. However as part of RCA I was trying to find the URL to close the loop. CDN came back with logs that exceeded 8190 bytes but if I see those URI they were around 4000+ char length only. |
|
Could this be an external denial-of-service attack? On Tue, Jan 26, 2016 at 1:51 PM, praveen262k notifications@github.com
|
|
Is it possible log the source URI in the error by pagespeed module? :) |
|
I think that's what got us into trouble at first -- we were logging it and On Tue, Jan 26, 2016 at 5:04 PM, praveen262k notifications@github.com
|
|
@jeffkaufman - Do you think you might be able to tell us a tentative release date at this point? Thanks! |
|
@praveen262k I'm sorry, we've been running into issues with some other changes we're planning to include in this same bugfix release. We're hoping to have a release out that fixes this issue within a week, but we can't promise anything. |
|
@jeffkaufman Understood! Thanks for the update! |
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
Creating an instaweb handler will run MakeRequestUrl, which assumes that request->unparsed_uri is non-null. So move the creation to after where we check that it's non-null. To be safe, move it all the way down to where it's first needed, in case some other validity checks end up being relevant. Fixes #1248
We enabled pagespeed module 2 weeks back. We have been monitoring site since then.
Last 2 days, we see below logs and server goes not responding. Any help would be appreciated! As of now we have pulled out the module from production.
GDB Backtrace log:
Apache log:
The text was updated successfully, but these errors were encountered: