Chrome occaisionally displays html code instead of parsing properly #340
Comments
ghost
assigned 
|
Can you paste your proxy configuration ? We had report some time ago of broken headers on apache 2.4. Maybe it is related |
|
It’s as basic as it gets really ProxyPass /static ! From: unbit [mailto:notifications@github.com] Can you paste your proxy configuration ? We had report some time ago of broken headers on apache 2.4. Maybe it is related — Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. |
|
i am trying to find a way to reproduce it. Do you have a way to report a tcpdump/wireshark/pcap (or generally the raw stream of bytes) sent by apache ? Thanks a lot |
|
Attached a capture. From: unbit [mailto:notifications@github.com] i am trying to find a way to reproduce it. Do you have a way to report a tcpdump/wireshark/pcap (or generally the raw stream of bytes) sent by apache ? Thanks a lot — Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. |
|
sorry i think github swallowed your attachment, try pasting it in the comment |
|
Hrm, well looks like the interface only allows for picture type attachments? It rejected my pcap file here too. If you give me your email address I can send it direct. |
|
roberto at unbit.it |
|
Here's a screen capture. |
|
is there any segmentation fault in apache error log ? |
|
No the error logs are clean both for the server and for the specific site. From: unbit [mailto:notifications@github.com] is there any segmentation fault in apache error log ? — Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. |
|
are you using the mpm "prefork" or the "worker" one ? |
|
Worker From: unbit [mailto:notifications@github.com] are you using the mpm "prefork" or the "worker" one ? — Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. |
|
Did not managed to reproduce it, but a customer confirmed he has the same bug. Can you try using SCGI instead of the uwsgi protocol ? If i rememebr correctly SCGI is natively supported in apache 2.4 (and works in the same way as mod_proxy_uwsgi). If SCGI is not available you can try FastCGI (or eventually HTTP even if you lose the ability to add request variables on the fly) |
|
FYI the mod_proxy_scgi module doesn’t seem to work properly. A co-worker patched it and submitted it to Apache to integrate, but we are also maintaining it here https://github.com/imsweb/mod-proxy-scgi in case anyone else might benefit from this. So far we haven’t been able to reproduce the issue when switching to scgi protocol, and once we got the proxy working as intended with these modifications it is a suitable solution. From: unbit [mailto:notifications@github.com] Did not managed to reproduce it, but a customer confirmed he has the same bug. Can you try using SCGI instead of the uwsgi protocol ? If i rememebr correctly SCGI is natively supported in apache 2.4 (and works in the same way as mod_proxy_uwsgi). If SCGI is not available you can try FastCGI (or eventually HTTP even if you lose the ability to add request variables on the fly) — Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. |
|
UPDATE: just received another report of the same bug, i still work on it |
|
Thanks for the update. Feel free to pass them that scgi module as a temp work around. Its been working great for us. -----Original Message----- UPDATE: just received another report of the same bug, i still work on it — Information in this e-mail may be confidential. It is intended only for the addressee(s) identified above. If you are not the addressee(s), or an employee or agent of the addressee(s), please note that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender of the error. |
|
Guys, sorry I'm bumping it, but there is no activity for 8 month already. Still experiencing the issues with Apache 2.2 / uwsgi 2.0 / mod_proxy_uwsgi |
|
the bug refers to Apache 2.4, there are no reports for 2.2, are you sure it is the same thing ? |
|
Yes, absolutely. #apt-cache policy apache2 libapache2-mod-proxy-uwsgi uwsgi | grep -B1 'Installed:'; uname -a Installed: 2.2.22-1ubuntu1.4libapache2-mod-proxy-uwsgi: Installed: 2.0-2uwsgi: And I've do some tcpdumping: Issue seems to be appearing only in dump1, thus uwsgi itself delivers content normally, screwing happens somewhere in mod_proxy_uwsgi / mod_proxy. Also, there is no strict relation to Chrome, under some circumstances other browsers are also getting their response headers screwed. Seems that Chrome just having more strict header parser and the issue appears much more frequently. Just in case, uswgi/2.0 package was created by myself (with linkage over apache/2.2) since there was no such version on Launchpad. I hope this will help. |
|
Are you able to report a whole (raw) response as generated by apache (read: as seen by the browser). This could help understanding what is going on. Thanks |
|
a new mod_proxy_uwsgi version have been committed (just clone the repo), maybe it fixes this problem too |
|
Sorry for a huge delay. As requested, raw response: As you can see, there are extra 16 bytes before actual HTTP header start which are leading to these issues. \r \n before the status line seems isn't seem to be correct either. Smells like the some sort of internal encapsulation is leaking outside of allowed borders... Any thoughs? |
|
but have you tried the latest mod_proxy_uwsgi from the git repo ? |
|
Sorry, not yet. Which repo/tag should I use? 2014-06-24 16:31 GMT+03:00 unbit notifications@github.com:
|
|
the master branch has latest code |
|
Tried with 2.0.6 today. Unfortunately no luck, issue still here... Here is how Chrome recognize it: GET /some/path/requested HTTP/0.9 HTTP/0.9 200 OK When request handled properly, Chrome shows it as follows: GET /some/path/requested HTTP/1.1 HTTP/1.1 200 OK |
|
are you sure you have rebuilt the latest mod_proxy_uwsgi apache module ? |
|
Absolutely, packaged all the bunch of uwsgi modules from 2.0.6.tar.gz and installed on some servers. !# md5sum mod_proxy_uwsgi.2.0.0.so mod_proxy_uwsgi.2.0.6.so !# ls -al mod_proxy_uwsgi.2.0.0.so mod_proxy_uwsgi.2.0.6.so However same size of different versions confusing a bit... |
|
Hi! I'm experiencing the same problem, but I know the cause and possible solution! :-) I'm running Apache 2.2 with mod_proxy_uwsgi. The problem is in duplicated zero-length-sized chunks when I'm not sure, whether the fault is in mod_proxy, mod_proxy_uwsi or uwsgi. PHP apps on the same server does not have the duplicate zero-sized chunks, so Apache is probably not causing it. Why @dene14 was able to reproduce it and @unibit no? Probably @dene14 is using Avast antivirus or other software not handling duplicated zero-sized chunks correctly. I spent last 2 weeks with Avast support to discover where the problem lies, because I managed to reproduce it ONLY on Windows XP ..... crazy. To prove my conclusions, try to run following two commands. They prints last lines of raw content of web communication. First is mod_proxy_uwsgi based, second is PHP based. Both on the same server. Notice the two zero-sized chunks at the end with first example. This is causing the error - last lines of HTTP communication generated by mod_proxy_uwsgi: Some open-source apps also have problem parsing these duplicate zero-sized chunks. It is somehow time-based, so it is not easy to trigger. For example latest Konqueror browser suffers from this issue. It is probably caused by some open-source library both Konqueror and Avast use, because the symptoms and timing constraints are identical. |
|
The tricky part here is that broken headers ('0\r\n' prepended) is caused by another application/library. I used Wireshark to dump raw data and confirmed '0\r\n' is not send by web server even when the error happened. It is added by some HTTP parsing library (or antivirus using the library), and the cause is those two zero-sized chunks. Software can by really unpredictable sometimes :-) |
|
This is crazy, you can exclude uWSGI from the equation as it sends raw output, so unless your app sends chunked frame there is nothing that can goes wrong. I assume the problem can be in mod_proxy_uwsgi (are you using latest version ? we improved in 2.0.6), but it does the same steps of mod_proxy_http. I'll dig a bit into apache sources. Btw there must be some kind of app-generated output that triggers this behaviour, as very few people reported it. |
|
Oh, obviously thanks for you report :) |
|
I'm using plain Django, so no chunking should be involved on Python side. Should I prepare some very basic testing Python app? Any links? I'm using mod_proxy_uwsgi somewhere around 2.0.4, but first chance to upgrade will be tonight (discovered the cause today morning). Did you verified that Apache + mod_proxy_uwsgi in your test env does not produce the duplicate-zero-chunks problem? |
|
Yes, no zero trailings, le me know how it goes with latest version, maybe it is enough. Now i am searching in httpd sources where (and when) the trailing chunk is added. |
|
I have a server with apache 2.2.27 and uwsgi 2.0.8 and it sometimes exhibits this issue as well. The curl command cuchac posted above seems to reproduce it pretty reliably. |
|
Damned Avast! Seems people from Czech Republic are most affected :-) They told me next release will fix this issue, so most of affected users will get fixed by next Avast upgrade. Latest change in mod_apache_uwsgi is from 2.0.6 release so it seems even git version will be affected. Is there anything we can do to help you solve this? I will try to upgrade tonight and make some tiny python app without any framework to test. My version of Apache is 2.2.27-r4, Gentoo distribution. |
|
Trying hard to reproduce the problem without luck. Can you give me the list of apache modules loaded and used by the www.wpj.cz instance ? |
|
I can for my instance: gives me the zeroes. This is also Gentoo system, the zeroes still come when the module list is reduced to this: |
|
Hi! and the problem is still there. I'm trying to reproduce the same issue on deveopment Gentoo box to avoid experimenting on production site. |
|
Hi!
|
|
I am pretty sure the problem is in this function: https://github.com/unbit/uwsgi/blob/master/apache2/mod_proxy_uwsgi.c#L301 can you try adding debug log all over the place to understand what is passed to apache ? the critical part is here: https://github.com/unbit/uwsgi/blob/master/apache2/mod_proxy_uwsgi.c#L368 |
|
Got it! :-)
I've added debug statements like this: for(;;) {
rv = ap_get_brigade(rp->input_filters, bb,
AP_MODE_READBYTES, mode,
conf->io_buffer_size);
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "READ");
apr_brigade_length(bb, 0, &readbytes);
if (mode == APR_NONBLOCK_READ && (APR_STATUS_IS_EAGAIN(rv)
|| (rv == APR_SUCCESS && (APR_BRIGADE_EMPTY(bb) /*|| readbytes == 0*/)))) {
e = apr_bucket_flush_create(c->bucket_alloc);
APR_BRIGADE_INSERT_TAIL(bb, e);
if (ap_pass_brigade(r->output_filters, bb) || c->aborted) {
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "aborted");
break;
}
apr_brigade_cleanup(bb);
mode = APR_BLOCK_READ;
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "APR_BLOCK_READ");
continue;
}
else if (rv == APR_EOF) {
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "APR_EOF");
break;
}
else if (rv != APR_SUCCESS) {
ap_proxy_backend_broke(r, bb);
ap_pass_brigade(r->output_filters, bb);
backend_broke = 1;
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "backend_broke");
break;
}
mode = APR_NONBLOCK_READ;
apr_brigade_length(bb, 0, &readbytes);
backend->worker->s->read += readbytes;
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "READ bytes: %d", readbytes);
if (APR_BRIGADE_EMPTY(bb)) {
apr_brigade_cleanup(bb);
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, rv, r, "CLEANUP");
break;
}
ap_proxy_buckets_lifetime_transform(r, bb, pass_bb);
ap_pass_brigade(r->output_filters, pass_bb);
apr_brigade_cleanup(bb);
apr_brigade_cleanup(pass_bb);
}and the resulting logs are for the correct result: and for incorrect result: After uncommenting |
|
Great work, i'll check why the other proxies do not do it and i will adapt your patch. I will release 2.0.9 as soon as it is ok. Thanks a lot |
|
My "patch" is not really a fix. I don't understand the code fully enough. There are possibly some side-effects I don't foresee. |
|
Yes, do not worry, now i know why, how and when the problem happens. I will push a fix asap |
|
Can you try with the pushed patch ? Thanks |
|
Sorry for such a delay. |
|
ok, i'll try to push another approach later, thanks again |
|
Another attempt e051e4f |
|
Hi, unfortunately the issue still persists. I've added debug output again. Here are results: Correct: Incorrect: Annotated code: int finish = 0;
while(!finish) {
rv = ap_get_brigade(rp->input_filters, bb,
AP_MODE_READBYTES, mode,
conf->io_buffer_size);
if (APR_STATUS_IS_EAGAIN(rv)
|| (rv == APR_SUCCESS && APR_BRIGADE_EMPTY(bb)) ) {
e = apr_bucket_flush_create(c->bucket_alloc);
APR_BRIGADE_INSERT_TAIL(bb, e);
if (ap_pass_brigade(r->output_filters, bb) || c->aborted) {
break;
}
apr_brigade_cleanup(bb);
mode = APR_BLOCK_READ;
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "mode = APR_BLOCK_READ");
continue;
}
else if (rv == APR_EOF) {
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "APR_EOF");
break;
}
else if (rv != APR_SUCCESS) {
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "rv != APR_SUCCESS");
ap_proxy_backend_broke(r, bb);
ap_pass_brigade(r->output_filters, bb);
backend_broke = 1;
break;
}
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "mode = APR_NONBLOCK_READ");
mode = APR_NONBLOCK_READ;
apr_brigade_length(bb, 0, &readbytes);
backend->worker->s->read += readbytes;
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "readbytes: %d", readbytes);
if (APR_BRIGADE_EMPTY(bb)) {
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "APR_BRIGADE_EMPTY");
apr_brigade_cleanup(bb);
break;
}
ap_proxy_buckets_lifetime_transform(r, bb, pass_bb);
// found the last brigade?
if (APR_BUCKET_IS_EOS(APR_BRIGADE_LAST(bb))){
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "APR_BUCKET_IS_EOS(APR_BRIGADE_LAST(bb))");
finish = 1;
}
if (ap_pass_brigade(r->output_filters, pass_bb) != APR_SUCCESS || c->aborted) {
ap_log_rerror(APLOG_MARK, APLOG_ERR, rv, r, "ap_pass_brigade(r->output_filters, pass_bb) != APR_SUCCESS || c->aborted");
finish = 1;
}
apr_brigade_cleanup(bb);
apr_brigade_cleanup(pass_bb);
}
e = apr_bucket_eos_create(c->bucket_alloc);
APR_BRIGADE_INSERT_TAIL(bb, e);The simplest fix I found was just to skip if (readbytes > 0)
if (ap_pass_brigade(r->output_filters, pass_bb) != APR_SUCCESS || c->aborted) {
.... |
|
what about this approach ? d69e234 |
|
Great, that seems to fix the issue. I've done several minutes of testing and not a single error. |
|
great, let's close this (again) |
It seems like on occaision (not every time and hard to reproduce) my python sites in chrome get a 0 before the html headers. Chrome chokes on the processing and displays the html code instead of processing the website. I'm assuming IE and Firefox coded around this and just process the html properly. I'm running Apache 2.4 with mod_proxy_uwsgi. I'm using uwsgi version 1.9.13.
The text was updated successfully, but these errors were encountered: