2.0.0b1 showing request headers

[rafaelss]

I just tried b1 in one of my applications and it's showing headers in the page

http://www.evernote.com/shard/s22/sh/9ac87138-1a86-4991-9b6b-4d95855eaf60/3b1bfae015d8779a37727a4050a597c3

update:

$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

[evanphx]

Could I get a little info about the app? I'll need to reproduce the problem. If you can at least tell me what requests are made from the page that cause the problem that should help. Also, which browser did you use?

[rafaelss]

weird. I can't reproduce this anymore. I'll keep using beta1 to see if the issue shows up again.

[kyledecot]

I'm also experiencing this w/ my Rails app. It doesn't happen on every page load though. It's very intermittent. Here are a couple of screenshots. As you can see sometimes the raw HTML w/ the headers is spit out and others it seems to inject the headers in the page page.

https://www.evernote.com/shard/s222/sh/0ac8606b-ca14-4a1c-b035-c90c672f7f87/80896e6d533206a8b29d5c564cc9290b

https://www.evernote.com/shard/s222/sh/de4a3520-84f5-4bc7-a588-a105c4a430da/570c451fdfefc5077c80e1e44cec0881

What information about my setup would be useful in tracking down the source of this? I'll start you off w/ my nginx config:

upstream production {
  server unix:///home/deployer/production/shared/sockets/puma.sock fail_timeout=0;
}

server {
    server_name theflowskatepark.com;
    rewrite ^(.*) http://www.theflowskatepark.com$1 permanent;
}

server {
  listen 80;
  server_name www.theflowskatepark.com;
  root /home/deployer/production/current/public;

  location ^~ /assets/ {
    gzip_static on;
    expires max;
    add_header Cache-Control public;
  }

  try_files $uri/index.html $uri @production;
  location @production {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_pass http://production;
  }

  error_page 500 502 503 504 /500.html;
  client_max_body_size 4G;
  keepalive_timeout 5;

}

[evanphx]

I don't suppose you'd let me run your rails app? 😸

I haven't yet been able to reproduce this and I've been through the request flow a few times and haven't been able to identify a place where it might happen.

If you can let me run your rails app, I can promise to do it in private if you don't wish it to be publicly available.

[kyledecot]

Certainly, though I haven't seen the issue when running my project locally. It only seems to happen on my linode server. I'm wondering if this is because locally I run rails s puma and on production I am running:

cd /home/deployer/production/current && RAILS_ENV=production bundle exec puma -d -b 'unix:///home/deployer/production/shared/sockets/puma.sock' -S /home/deployer/production/shared/sockets/puma.state --control 'unix:///home/deployer/production/shared/sockets/pumactl.sock' >> /home/deployer/production/shared/log/puma-production.log 2>&1

In any case my project is located at https://github.com/kyledecot/the-flow-skatepark.

[duyleekun]

Can you share the safari html source of the page when that happens? It may help pinpoint the issue

[kyledecot]

Case #1
Renders somewhat properly but the headers seems to be displayed in within the page

https://www.evernote.com/shard/s222/sh/0ac8606b-ca14-4a1c-b035-c90c672f7f87/80896e6d533206a8b29d5c564cc9290b
http://snipt.org/vgik0

Case #2
Raw HTML rather then a rendered page

https://www.evernote.com/shard/s222/sh/de4a3520-84f5-4bc7-a588-a105c4a430da/570c451fdfefc5077c80e1e44cec0881
http://snipt.org/vgil1

I should also note that case #1 is much easier to reproduce (perhaps 1/5 page loads). Case #2 might only only happen every 20 page loads.

[kyledecot]

Any progress on this issue?

[haberbyte]

I have the same issue.

I hope we can reproduce it so @evanphx has a place to look into.

What i found so far is that it seems to happen when binding to a unix socket.
I'm using JRuby 1.7.0, so that might play a role, too.

[haberbyte]

OK I have put together a very small hello world rack app that demonstrates the problem.

Have a look here: https://github.com/habermann24/pumabug

To really reproduce it you should try listening on a socket and use nginx.
Sometimes you will see the "Hello, World!" text appear, other times the status code "HTTP/1.0 200".

The more text you add to config.ru's body output, the more HTTP header you will get :)

I explicitly said "and use nginx" because this doesn't happen if you forward a TCP port to that socket with socat like this:
socat TCP-LISTEN:1234,reuseaddr UNIX-CLIENT:/tmp/puma.sock

update:

interesting discovery: if i tell puma to use n minimum threads, then on the n+1's request i get the header.

For example:

bundle exec puma config.ru -b unix:///tmp/puma.sock -t 4:8 would mean after the 4th request i get the header displayed

[haberbyte]

Sorry about the number of posts, but i think i found a solution to the bug.

nginx uses HTTP 1.0 by default when proxying requests.
When i configure nginx to use HTTP 1.1 as suggested in the keepalive section in the docs, the weird behavior of puma is gone, see http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

So one needs to add the following to the nginx config:

location @app {
   ...
   proxy_http_version 1.1;
   proxy_set_header Connection "";
   ...
}

I have no idea why this works yet and why it doesn't without those settings.
But i assume puma expects HTTP 1.1 and there is some trouble when nginx talks 1.0...
Hopefully this will give people who know what they're doing a hint at how to fix this in puma to properly handle HTTP 1.0.

Meanwhile i'm going to do some reading about the topic ;-)

[evanphx]

I found the bug. Will have a new release out today.

• Evan // via iPhone

On Nov 16, 2012, at 1:57 PM, Jan notifications@github.com wrote:

Sorry about the number of posts, but i think i found a solution to the bug.

nginx uses HTTP 1.0 by default when proxying requests.
When i configure nginx to use HTTP 1.1 as suggested in the keepalive section in the docs, the weird behavior of puma is gone, see http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

So one needs to add the following to the nginx config:

location @app {
...
proxy_http_version 1.1;
proxy_set_header Connection "";
...
}
I have no idea why this works yet and why it doesn't without those settings.
But i assume puma expects HTTP 1.1 and there is some trouble when nginx talks 1.0...
Hopefully this will give people who know what they're doing a hint at how to fix this in puma to properly handle HTTP 1.0.

Meanwhile i'm going to do some reading about the topic ;-)

—
Reply to this email directly or view it on GitHub.

[haberbyte]

That is awesome!!!

My guess is it is exactly what's stated here under "The connection header":

http://www2.research.att.com/~bala/papers/h0vh1.html

Puma needs to remove the "hop-by-hop" headers that nginx sends, or something like that :-)

On 16.11.2012, at 20:16, Evan Phoenix notifications@github.com wrote:

I found the bug. Will have a new release out today.

• Evan // via iPhone

On Nov 16, 2012, at 1:57 PM, Jan notifications@github.com wrote:

Sorry about the number of posts, but i think i found a solution to the bug.

nginx uses HTTP 1.0 by default when proxying requests.
When i configure nginx to use HTTP 1.1 as suggested in the keepalive section in the docs, the weird behavior of puma is gone, see http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

So one needs to add the following to the nginx config:

location @app {
...
proxy_http_version 1.1;
proxy_set_header Connection "";
...
}
I have no idea why this works yet and why it doesn't without those settings.
But i assume puma expects HTTP 1.1 and there is some trouble when nginx talks 1.0...
Hopefully this will give people who know what they're doing a hint at how to fix this in puma to properly handle HTTP 1.0.

Meanwhile i'm going to do some reading about the topic ;-)

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub.