New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REFUSED_STREAM for http/2 POST to nginx #1040

Closed
jeroen opened this Issue Sep 28, 2016 · 14 comments

Comments

Projects
None yet
3 participants
@jeroen
Contributor

jeroen commented Sep 28, 2016

We are working on a new API based on express.js behind nginx. Everything works for HTTP1, but for (lib)curl client which are built --with-nghttp2 any POST request fails with CURLE_HTTP2_STREAM.

Below a reproducible example that fails with libcurl 7.50.3 (--with-nghttp2)

curl -v https://builder.r-hub.io/api/check/validate_email \
 -H 'Content-Type: application/json' \
 -d '{"email":"somebody@gmail.com"}'

Example output: https://git.io/vPfIS Is this a bug in libcurl or a nginx config problem?

@tatsuhiro-t @gaborcsardi

@bagder

This comment has been minimized.

Show comment
Hide comment
@bagder

bagder Sep 28, 2016

Member

This sounds like a suspicious nginx issue. What nginx version are you running this against?

Member

bagder commented Sep 28, 2016

This sounds like a suspicious nginx issue. What nginx version are you running this against?

@bagder bagder added the HTTP/2 label Sep 28, 2016

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

nginx version: nginx/1.10.1

ubuntu@ip:~$ dpkg -l | grep nginx
ii  nginx                               1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server
ii  nginx-common                        1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server - common files
ii  nginx-full                          1.10.1-0+trusty0                 amd64        nginx web/proxy server (standard version)

I can also share the config if you need it.

gaborcsardi commented Sep 28, 2016

nginx version: nginx/1.10.1

ubuntu@ip:~$ dpkg -l | grep nginx
ii  nginx                               1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server
ii  nginx-common                        1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server - common files
ii  nginx-full                          1.10.1-0+trusty0                 amd64        nginx web/proxy server (standard version)

I can also share the config if you need it.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

The web apps are managed by dokku (https://github.com/dokku/dokku), which generated the nginx config, and supposedly auto-detects if nginx supports http/2, based on version numbers: https://github.com/dokku/dokku/blob/cb9c5b6988b9c8471cf5ca764d9c361af03327f0/plugins/nginx-vhosts/functions#L151

gaborcsardi commented Sep 28, 2016

The web apps are managed by dokku (https://github.com/dokku/dokku), which generated the nginx config, and supposedly auto-detects if nginx supports http/2, based on version numbers: https://github.com/dokku/dokku/blob/cb9c5b6988b9c8471cf5ca764d9c361af03327f0/plugins/nginx-vhosts/functions#L151

@bagder

This comment has been minimized.

Show comment
Hide comment
@bagder

bagder Sep 28, 2016

Member

I think this is the same flaw discussed for example here: square/okhttp#2506

It is apparently fixed in nginx 1.11 and their changelog seems to mention it.

Member

bagder commented Sep 28, 2016

I think this is the same flaw discussed for example here: square/okhttp#2506

It is apparently fixed in nginx 1.11 and their changelog seems to mention it.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

That's indeed plausible. Let me see if I can upgrade nginx. It might take a day or two.

gaborcsardi commented Sep 28, 2016

That's indeed plausible. Let me see if I can upgrade nginx. It might take a day or two.

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 28, 2016

Contributor

@gaborcsardi you can probably upgrade via the nginx ubuntu PPA:

sudo add-apt-repository nginx/development
sudo apt-get update
sudo apt-get dist-upgrade
Contributor

jeroen commented Sep 28, 2016

@gaborcsardi you can probably upgrade via the nginx ubuntu PPA:

sudo add-apt-repository nginx/development
sudo apt-get update
sudo apt-get dist-upgrade
@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

@jeroenooms Thanks!

Just for the record, I did:

sudo add-apt-repository ppa:nginx/development
sudo apt-get update
sudo apt-get install nginx nginx-common nginx-full
sudo service nginx restart

and then everything is fine! Thanks all for the help!

gaborcsardi commented Sep 28, 2016

@jeroenooms Thanks!

Just for the record, I did:

sudo add-apt-repository ppa:nginx/development
sudo apt-get update
sudo apt-get install nginx nginx-common nginx-full
sudo service nginx restart

and then everything is fine! Thanks all for the help!

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 28, 2016

Contributor

OK I think this is solved. Perhaps communicate to the dokku folks that they should not enable http/2 for nginx 1.10?

Contributor

jeroen commented Sep 28, 2016

OK I think this is solved. Perhaps communicate to the dokku folks that they should not enable http/2 for nginx 1.10?

@jeroen jeroen closed this Sep 28, 2016

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

@jeroenooms Good idea. Will do.

gaborcsardi commented Sep 28, 2016

@jeroenooms Good idea. Will do.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Oct 1, 2016

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

gaborcsardi commented Oct 1, 2016

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen May 15, 2017

Contributor

The nginx fix has landed into nginx stable (1.12.0). On Ubuntu backports are available from the ppa:

sudo add-apt-repository nginx/stable
sudo apt-get update
sudo apt-get dist-upgrade

This fixed the problem on my servers.

Contributor

jeroen commented May 15, 2017

The nginx fix has landed into nginx stable (1.12.0). On Ubuntu backports are available from the ppa:

sudo add-apt-repository nginx/stable
sudo apt-get update
sudo apt-get dist-upgrade

This fixed the problem on my servers.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi commented May 15, 2017

OK.

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen May 16, 2017

Contributor

@bagder could you consider adding a workaround in curl for this nginx behavior? Even though it is a bug in nginx, no other browser or http/2 client trips over it; the request only fails with libcurl.

It would be great if curl could handle this a bit more gracefully (like browsers) because the 'buggy' version of nginx will be around for many more years.

Contributor

jeroen commented May 16, 2017

@bagder could you consider adding a workaround in curl for this nginx behavior? Even though it is a bug in nginx, no other browser or http/2 client trips over it; the request only fails with libcurl.

It would be great if curl could handle this a bit more gracefully (like browsers) because the 'buggy' version of nginx will be around for many more years.

@bagder

This comment has been minimized.

Show comment
Hide comment
@bagder

bagder May 18, 2017

Member

could you consider adding a workaround in curl for this nginx behavior?

I would certainly not mind handling the refused stream case better than we do right now, but it isn't clear to me exactly how we can mitigate this nginx issue without sacrificing some functionality or performance when working with standard conforming servers.

The okhttp mitigation (square/okhttp#2543) is per the description mostly assuming that if a stream is retried due to this nginx error, the server has also had enough time to ack the settings properly so that the next attempt will work. A plausible assumption I suppose.

Does anyone still have a server / use case setup that reproduces this issue?

Member

bagder commented May 18, 2017

could you consider adding a workaround in curl for this nginx behavior?

I would certainly not mind handling the refused stream case better than we do right now, but it isn't clear to me exactly how we can mitigate this nginx issue without sacrificing some functionality or performance when working with standard conforming servers.

The okhttp mitigation (square/okhttp#2543) is per the description mostly assuming that if a stream is retried due to this nginx error, the server has also had enough time to ack the settings properly so that the next attempt will work. A plausible assumption I suppose.

Does anyone still have a server / use case setup that reproduces this issue?

@lock lock bot locked as resolved and limited conversation to collaborators May 6, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.