New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strip Connection and other hop-by-hop headers when saving input resource, because the underlying server will add a new Connection header. #664

Closed
GoogleCodeExporter opened this Issue Apr 6, 2015 · 22 comments

Comments

Projects
None yet
1 participant
@GoogleCodeExporter
Copy link

GoogleCodeExporter commented Apr 6, 2015

Received the following error on several CSS files:
Error 330 (net::ERR_CONTENT_DECODING_FAILED): Unknown error.

Receive various javascript "could not load" or parsing errors (using Chrome 
console):
Uncaught Error: Could not load 'cms.QLButton'; last tried '/js/cms/QLButton.js'
Error
Error parsing in _ContentSetter#Setter_pf_ui_dashboard_Widget_0_pane_0 
Error
 /fission/js/dojo/dojo/../dijit/layout/ContentPane.js:525
Error undefined running custom onLoad code: This deferred has already been 
resolved 

Interestingly enough the javascript parse errors are not corrected when adding 
the URL parameter: "ModPagespeed=off", but work fine when I completely shut 
mod_pagespeed off.

These errors occur only on our administrative panel, so they aren't easy to 
provide specific examples, but they basically occur on every page load & render 
our admin panel completely unusable.

What version of the product are you using (please check X-Mod-Pagespeed
header)?
X-Mod-Pagespeed: 1.4.0.0-2742

On what operating system?
Arch Linux

Which version of Apache?
Apache/2.2.24

Which MPM?
Prefork

URL of broken page:
http://www.healthlinksgolf.org/fission/js/dojo/dijit/themes/claro/TitlePane_rtl.
css (this is the example URL of one the regularly receives the 330 content 
decoding error, but it will not currently break since I've been force to turn 
off mod_pagespeed on this server)


Original issue reported on code.google.com by mwhe...@360psg.com on 4 Apr 2013 at 8:15

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I'll take a look at this

Original comment by jmara...@google.com on 11 Apr 2013 at 12:55

  • Changed state: Accepted
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Recently upgraded to Mod Pagespeed to 1.7.0.0-3389, and I'm again seeing 
similar issues again sporadically.

Unfortunately the original website is no longer active, but today I'm regularly 
able to replicate using http://www.sushashoes.com (admin panel).

I'm getting 330 errors on various CSS/JS files, again very sporadically 
(different files each load).

When I can replicate the error direct in a browser I get the following:

Error 330 (net::ERR_CONTENT_DECODING_FAILED): Unknown error.

An example of some of the files this has occurred on are as follows:

http://www.sushashoes.com/fission/js/dojo/dojox/layout/resources/GridContainer.c
ss
http://www.sushashoes.com/fission/js/dojo/dojox/mdnd/resources/dnd.css
http://www.sushashoes.com/fission/js/dojo/dijit/icons/editorIcons.css

Not even sure what other information I could provide to help diagnose. Please 
let me know.

Original comment by mwhe...@360psg.com on 22 Aug 2013 at 6:54

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I'm even seeing these issues when I include "?ModPagespeed=off" within my query 
string. It appears as if this just prevents the HTML from processing, but the 
CSS files (perhaps JS too) are still being minified, or otherwise processed by 
mod_pagespeed.

When I outright turn off Mod Pagespeed, the issues immediately cease to be an 
issue.

Original comment by mwhe...@360psg.com on 22 Aug 2013 at 7:10

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Do you have ModPagespeedInPlaceResourceOptimization on?

Original comment by jmara...@google.com on 22 Aug 2013 at 7:38

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I've convinced myself that you have ModPagespeedInPlaceResourceOptimization on.

http://www.sushashoes.com/fission/js/script.aculo.us/scriptaculous-js-1.8.2/effe
cts.js?ModPagespeed=off produces nicely formatted JavaScript.

http://www.sushashoes.com/fission/js/script.aculo.us/scriptaculous-js-1.8.2/effe
cts.js produces minified JavaScript.

When you add ?ModPagespeed=off to the HTML URL, that means mod_pagespeed won't 
rewrite the HTML, but the query-param does not get transmitted to the server 
for the JS request.  So that feature does not fully turn off MPS.  You can set 
it in a request header, which will be transmitted to the server, disabling 
in-place resource optimization.

I have reproduced the problem (easily) in Chrome Dev Tools.  It looks to me 
like 4 of the 5 or so of the uncombined scriptalicious files are failing to 
load, and they have strange response headers.

One thing that's weird is this header: Connection:Keep-Alive, Keep-Alive.  That 
header is on all the failed JS files, but it's also on one of the ones that 
worked.

I don't know why the "Keep-Alive" is repeated or whether that would be a 
problem.

Original comment by jmara...@google.com on 22 Aug 2013 at 7:53

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

It appears as if I do have ModPagespeedInPlaceResourceOptimization on. This 
wasn't intentional, looks like it was on as a part of the default config 
shipped with the product.

Do you suggest turning this off?

Original comment by mwhe...@360psg.com on 22 Aug 2013 at 7:55

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I don't see this feature in the default template.  I also think it's a good 
feature, and I wasn't aware of this problem before.

I don't know for sure if it's related to 
ModPagespeedInPlaceResourceOptimization, but it's worth turning it off to see 
if that fixes the problem.

Original comment by jmara...@google.com on 22 Aug 2013 at 7:57

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I turned it off, and haven't yet experienced the same issue again. I think this 
may have been the issue, potentially due to mismatched Content-Length headers 
(supplied unminified content lengths with minified content, and vice versa)?

This is all speculation however, since it was very sporadic, it was difficult 
to reproduce (wasn't occurring with the same source code on different websites).

Original comment by mwhe...@360psg.com on 22 Aug 2013 at 8:00

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Are you adding any headers in your apache configuration?  What does your 
keep-alive setting look like?

Original comment by jmara...@google.com on 22 Aug 2013 at 8:14

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

1. It looks like on compilation, debug.conf.template is being appended to 
pagespeed.conf.template to create pagespeed.conf. Perhaps I should eliminate 
anything generated from debug.conf.template (which is where the 
ModPagespeedInPlaceResourceOptimization directive came from).

2. Our default Apache configuration for Keep Alive is as follows:
httpd-default.conf:
  16: KeepAlive On
  23: MaxKeepAliveRequests 100
  29: KeepAliveTimeout 5

Though, we override the KeepAliveTimeout directive to be 2, as follows:
   5: KeepAliveTimeout 2

I'm not seeing that header repeat like you provided. I'm receiving the 
following headers on the above mentioned effects.js:


Date: Thu, 22 Aug 2013 20:23:29 GMT
Server: Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e DAV/2
Last-Modified: Tue, 02 Apr 2013 18:07:30 GMT
Accept-Ranges: bytes
Cache-Control: public, max-age=600
Expires: Thu, 29 Aug 2013 20:23:29 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
X-Extra-Header: 1
Content-Length: 8782
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: application/javascript

Original comment by mwhe...@360psg.com on 22 Aug 2013 at 8:25

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

debug.conf.template is not intended for production.  It's needed to set up the 
vhosts for running install/apache_system_test.sh.

I don't see the double "Keep-Alive" either now.  That must be caused by 
InPlaceResourceOptimization.  It isn't purposefully adding that header itself, 
of course, but it might be doubled because of the way the origin fetch is 
captured via loopback.  Maybe we are transmitting origin-headers that we 
shouldn't.

Please leave InPlaceResourceOptimization off for now, and we'll investigate.


In general, though, I'd clean out the contribution from debug.conf.template 
from your pagespeed.conf.

Original comment by jmara...@google.com on 22 Aug 2013 at 8:36

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Summary was: Random & Systematic JS/CSS Loading Errors

To repro, probably we'll want to have:

  KeepAlive On
  MaxKeepAliveRequests 100
  KeepAliveTimeout 5

in our test confguration.


Original comment by jmara...@google.com on 22 Aug 2013 at 8:37

  • Changed title: Error 330 (net::ERR_CONTENT_DECODING_FAILED): Unknown error on resources when InPlaceResourceOptimization is enabled.
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

It does sound plausible that InPlaceResourceRecorder might sometimes capture 
transport headers like Connection: or even Length:.

Hmm, perhaps that could also cause the "no keep-alive" problem someone reported 
on mod-pagespeed-discuss? 

Original comment by morlov...@google.com on 22 Aug 2013 at 8:47

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Reproduced.

Added to pagespeed.conf:
     KeepAlive On
     MaxKeepAliveRequests 100
     KeepAliveTimeout 5
restarted debug apache & then typed:

% wget -q --save-headers -O - 
http://localhost:8080/mod_pagespeed_example/styles/blue.css | grep Connection
Connection: Keep-Alive
% wget -q --save-headers -O - 
http://localhost:8080/mod_pagespeed_example/styles/blue.css | grep Connection
Connection: Keep-Alive, Keep-Alive

I think we should not keep the origin headers at all for ipro responses.  Note 
that we do *not* have this problem for .pagespeed. responses:

% wget -q --save-headers -O - 
http://localhost:8080/mod_pagespeed_example/styles/bold.css.pagespeed.ce.GeAD1d5
ga5.css|grep Connection
Connection: Keep-Alive

That happens no matter how many times I repeat.

Original comment by jmara...@google.com on 22 Aug 2013 at 9:36

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Original comment by jmara...@google.com on 22 Aug 2013 at 9:37

  • Changed state: Started
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I don't think it's as simple as not keeping the origin headers, since we want 
to keep whatever the handler produced, just not what the filters between the 
handler and the recorder produced. Perhaps the recorder is just running too 
late?

Original comment by morlov...@google.com on 22 Aug 2013 at 9:39

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

[deleted comment]
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

It looks like you are building from source.  If I gave you a patch file, would 
you be willing to rebuild mod_pagespeed.so and turn on 
InPlaceResourceOptimization again, to see if this problem reproduces?

Thanks!

Original comment by jmara...@google.com on 22 Aug 2013 at 10:48

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

here is a patch which I believe will fix the problem.

Original comment by jmara...@google.com on 22 Aug 2013 at 10:52

Attachments:

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

I have a new alternate patch which I believe is a more complete fix to the 
problem.  This patch alone will not be committed to trunk as it (a) has not 
been reviewed yet and (b) introduces a small performance regression.  Another 
couple of outstanding changes: one committed to trunk already, and one pending 
review, will eliminate the performance penalty.

I think the penalty will be small though, and it should be worth trying to see 
if it resolves the extra Connection header issues.

Original comment by jmara...@google.com on 5 Sep 2013 at 12:22

Attachments:

@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Fixed in https://code.google.com/p/modpagespeed/source/detail?r=3480

Original comment by jmara...@google.com on 9 Sep 2013 at 5:21

  • Changed state: Fixed
  • Added labels: Milestone-v30, release-note
@GoogleCodeExporter

This comment has been minimized.

Copy link

GoogleCodeExporter commented Apr 6, 2015

Update the summary for better clarifying the issue.

Original comment by hui...@google.com on 28 Oct 2013 at 3:01

  • Changed title: Strip Connection and other hop-by-hop headers when saving input resource, because the underlying server will add a new Connection header.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment