REFUSED_STREAM for http/2 POST to nginx

[jeroen]

We are working on a new API based on express.js behind nginx. Everything works for HTTP1, but for (lib)curl client which are built --with-nghttp2 any POST request fails with CURLE_HTTP2_STREAM.

Below a reproducible example that fails with libcurl 7.50.3 (--with-nghttp2)

curl -v https://builder.r-hub.io/api/check/validate_email \
 -H 'Content-Type: application/json' \
 -d '{"email":"somebody@gmail.com"}'

Example output: https://git.io/vPfIS Is this a bug in libcurl or a nginx config problem?

@tatsuhiro-t @gaborcsardi

[bagder]

This sounds like a suspicious nginx issue. What nginx version are you running this against?

[gaborcsardi]

nginx version: nginx/1.10.1

ubuntu@ip:~$ dpkg -l | grep nginx
ii  nginx                               1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server
ii  nginx-common                        1.10.1-0+trusty0                 all          small, powerful, scalable web/proxy server - common files
ii  nginx-full                          1.10.1-0+trusty0                 amd64        nginx web/proxy server (standard version)

I can also share the config if you need it.

[gaborcsardi]

The web apps are managed by dokku (https://github.com/dokku/dokku), which generated the nginx config, and supposedly auto-detects if nginx supports http/2, based on version numbers: https://github.com/dokku/dokku/blob/cb9c5b6988b9c8471cf5ca764d9c361af03327f0/plugins/nginx-vhosts/functions#L151

[bagder]

I think this is the same flaw discussed for example here: square/okhttp#2506

It is apparently fixed in nginx 1.11 and their changelog seems to mention it.

[gaborcsardi]

That's indeed plausible. Let me see if I can upgrade nginx. It might take a day or two.

[jeroen]

@gaborcsardi you can probably upgrade via the nginx ubuntu PPA:

sudo add-apt-repository nginx/development
sudo apt-get update
sudo apt-get dist-upgrade

[gaborcsardi]

@jeroenooms Thanks!

Just for the record, I did:

sudo add-apt-repository ppa:nginx/development
sudo apt-get update
sudo apt-get install nginx nginx-common nginx-full
sudo service nginx restart

and then everything is fine! Thanks all for the help!

[jeroen]

OK I think this is solved. Perhaps communicate to the dokku folks that they should not enable http/2 for nginx 1.10?

[gaborcsardi]

@jeroenooms Good idea. Will do.

[gaborcsardi]

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

[jeroen]

The nginx fix has landed into nginx stable (1.12.0). On Ubuntu backports are available from the ppa:

sudo add-apt-repository nginx/stable
sudo apt-get update
sudo apt-get dist-upgrade

This fixed the problem on my servers.

[gaborcsardi]

OK.

[jeroen]

@bagder could you consider adding a workaround in curl for this nginx behavior? Even though it is a bug in nginx, no other browser or http/2 client trips over it; the request only fails with libcurl.

It would be great if curl could handle this a bit more gracefully (like browsers) because the 'buggy' version of nginx will be around for many more years.

[bagder]

could you consider adding a workaround in curl for this nginx behavior?

I would certainly not mind handling the refused stream case better than we do right now, but it isn't clear to me exactly how we can mitigate this nginx issue without sacrificing some functionality or performance when working with standard conforming servers.

The okhttp mitigation (square/okhttp#2543) is per the description mostly assuming that if a stream is retried due to this nginx error, the server has also had enough time to ack the settings properly so that the next attempt will work. A plausible assumption I suppose.

Does anyone still have a server / use case setup that reproduces this issue?