Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REFUSED_STREAM for http/2 POST to nginx #1040

Closed
jeroen opened this issue Sep 28, 2016 · 14 comments
Closed

REFUSED_STREAM for http/2 POST to nginx #1040

jeroen opened this issue Sep 28, 2016 · 14 comments
Labels

Comments

@jeroen
Copy link
Contributor

@jeroen jeroen commented Sep 28, 2016

We are working on a new API based on express.js behind nginx. Everything works for HTTP1, but for (lib)curl client which are built --with-nghttp2 any POST request fails with CURLE_HTTP2_STREAM.

Below a reproducible example that fails with libcurl 7.50.3 (--with-nghttp2)

curl -v https://builder.r-hub.io/api/check/validate_email \
 -H 'Content-Type: application/json' \
 -d '{"email":"somebody@gmail.com"}'

Example output: https://git.io/vPfIS Is this a bug in libcurl or a nginx config problem?

@tatsuhiro-t @gaborcsardi

@bagder
Copy link
Member

@bagder bagder commented Sep 28, 2016

This sounds like a suspicious nginx issue. What nginx version are you running this against?

@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented Sep 28, 2016

nginx version: nginx/1.10.1

ubuntu@ip:~$ dpkg -l | grep nginx
ii  nginx                               1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server
ii  nginx-common                        1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server - common files
ii  nginx-full                          1.10.1-0+trusty0                 amd64        nginx web/proxy server (standard version)

I can also share the config if you need it.

@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented Sep 28, 2016

The web apps are managed by dokku (https://github.com/dokku/dokku), which generated the nginx config, and supposedly auto-detects if nginx supports http/2, based on version numbers: https://github.com/dokku/dokku/blob/cb9c5b6988b9c8471cf5ca764d9c361af03327f0/plugins/nginx-vhosts/functions#L151

@bagder
Copy link
Member

@bagder bagder commented Sep 28, 2016

I think this is the same flaw discussed for example here: square/okhttp#2506

It is apparently fixed in nginx 1.11 and their changelog seems to mention it.

@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented Sep 28, 2016

That's indeed plausible. Let me see if I can upgrade nginx. It might take a day or two.

@jeroen
Copy link
Contributor Author

@jeroen jeroen commented Sep 28, 2016

@gaborcsardi you can probably upgrade via the nginx ubuntu PPA:

sudo add-apt-repository nginx/development
sudo apt-get update
sudo apt-get dist-upgrade
@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented Sep 28, 2016

@jeroenooms Thanks!

Just for the record, I did:

sudo add-apt-repository ppa:nginx/development
sudo apt-get update
sudo apt-get install nginx nginx-common nginx-full
sudo service nginx restart

and then everything is fine! Thanks all for the help!

@jeroen
Copy link
Contributor Author

@jeroen jeroen commented Sep 28, 2016

OK I think this is solved. Perhaps communicate to the dokku folks that they should not enable http/2 for nginx 1.10?

@jeroen jeroen closed this Sep 28, 2016
@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented Sep 28, 2016

@jeroenooms Good idea. Will do.

@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented Oct 1, 2016

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

@jeroen
Copy link
Contributor Author

@jeroen jeroen commented May 15, 2017

The nginx fix has landed into nginx stable (1.12.0). On Ubuntu backports are available from the ppa:

sudo add-apt-repository nginx/stable
sudo apt-get update
sudo apt-get dist-upgrade

This fixed the problem on my servers.

@gaborcsardi
Copy link

@gaborcsardi gaborcsardi commented May 15, 2017

OK.

@jeroen
Copy link
Contributor Author

@jeroen jeroen commented May 16, 2017

@bagder could you consider adding a workaround in curl for this nginx behavior? Even though it is a bug in nginx, no other browser or http/2 client trips over it; the request only fails with libcurl.

It would be great if curl could handle this a bit more gracefully (like browsers) because the 'buggy' version of nginx will be around for many more years.

@bagder
Copy link
Member

@bagder bagder commented May 18, 2017

could you consider adding a workaround in curl for this nginx behavior?

I would certainly not mind handling the refused stream case better than we do right now, but it isn't clear to me exactly how we can mitigate this nginx issue without sacrificing some functionality or performance when working with standard conforming servers.

The okhttp mitigation (square/okhttp#2543) is per the description mostly assuming that if a stream is retried due to this nginx error, the server has also had enough time to ack the settings properly so that the next attempt will work. A plausible assumption I suppose.

Does anyone still have a server / use case setup that reproduces this issue?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.