New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not allow http2 for nginx 1.10.x #2435

Closed
gaborcsardi opened this Issue Sep 28, 2016 · 15 comments

Comments

Projects
None yet
3 participants
@gaborcsardi

gaborcsardi commented Sep 28, 2016

Description of problem:

dokku turns on http2 for nginx 1.10.x and above. But this version of nginx has a broken http2 implementation, and http2 aware clients fail to connect to it. Here is an example case: curl/curl#1040

nginx 1.11.x seems to be fine.

Output of the following commands

  • uname -a:
Linux ip-172-31-16-56 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • free -m:
             total       used       free     shared    buffers     cached
Mem:          3952       3260        691          6        318       1369
-/+ buffers/cache:       1571       2380
Swap:            0          0          0
  • docker version:
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64
  • docker -D info:
Containers: 28
 Running: 12
 Paused: 0
 Stopped: 16
Images: 151
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 316
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: ip-172-31-16-56
ID: 2RSZ:VYWR:Z577:KLV4:DVHJ:AHSQ:656I:FBOG:XSHB:FOJS:PXUD:TKZU
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
  • sigil -v:
0.4.0
  • docker run -ti gliderlabs/herokuish:latest herokuish version:
herokuish: 0.3.18
buildpacks:
  heroku-buildpack-multi     v1.0.0
  heroku-buildpack-ruby      v146
  heroku-buildpack-nodejs    v91
  heroku-buildpack-clojure   v75
  heroku-buildpack-python    v81
  heroku-buildpack-java      v44
  heroku-buildpack-gradle    v17
  heroku-buildpack-grails    v21
  heroku-buildpack-scala     v72
  heroku-buildpack-play      v26
  heroku-buildpack-php       v109
  heroku-buildpack-go        v44
  heroku-buildpack-erlang    fa17af9
  buildpack-nginx            v5
  emberjs.tgz                latest
  • dokku version:
0.7.1
  • dokku plugin:
plugn: dev
  00_dokku-standard    0.7.1 enabled    dokku core standard plugin
  20_events            0.7.1 enabled    dokku core events logging plugin
  apps                 0.7.1 enabled    dokku core apps plugin
  build-env            0.7.1 enabled    dokku core build-env plugin
  certs                0.7.1 enabled    dokku core certificate management plugin
  checks               0.7.1 enabled    dokku core checks plugin
  common               0.7.1 enabled    dokku core common plugin
  config               0.7.1 enabled    dokku core config plugin
  docker-options       0.7.1 enabled    dokku core docker-options plugin
  domains              0.7.1 enabled    dokku core domains plugin
  enter                0.7.1 enabled    dokku core enter plugin
  git                  0.7.1 enabled    dokku core git plugin
  jenkins              1.0.0 enabled    dokku jenkins service plugin with R installed
  logs                 0.7.1 enabled    dokku core logs plugin
  named-containers     0.7.1 enabled    dokku core named containers plugin
  nginx-vhosts         0.7.1 enabled    dokku core nginx-vhosts plugin
  plugin               0.7.1 enabled    dokku core plugin plugin
  proxy                0.7.1 enabled    dokku core proxy plugin
  ps                   0.7.1 enabled    dokku core ps plugin
  rabbitmq             1.0.0 enabled    dokku rabbitmq service plugin
  redirect             0.4.0 enabled    Simple redirects for apps
  redis                1.0.0 enabled    dokku redis service plugin
  repo                 0.7.1 enabled    dokku core repo plugin
  shell                0.7.1 enabled    dokku core shell plugin
  ssh-keys             0.7.1 enabled    dokku core ssh-keys plugin
  storage              0.7.1 enabled    dokku core storage plugin
  tags                 0.7.1 enabled    dokku core tags plugin
  tar                  0.7.1 enabled    dokku core tar plugin
  • cat /home/dokku/<app>/nginx.conf (if applicable):

server {
  listen      [::]:80;
  listen      80;
  server_name r-hub.io;
  access_log  off;
  return 301  $scheme://log.r-hub.io$request_uri;
}

server {
  listen      [::]:443 ssl http2;
  listen      443 ssl http2;
  server_name r-hub.io;
  access_log  off;

  ssl_certificate     /home/dokku/builder/tls/server.crt;
  ssl_certificate_key /home/dokku/builder/tls/server.key;

  return 301  $scheme://log.r-hub.io$request_uri;
}


server {
  listen      [::]:80;
  listen      80;
  server_name builder.r-hub.io;
  access_log  /var/log/nginx/builder-access.log;
  error_log   /var/log/nginx/builder-error.log;

  return 301 https://$host:443$request_uri;

}

server {
  listen      [::]:443 ssl http2;
  listen      443 ssl http2;
  server_name builder.r-hub.io;
  server_name builder.r-hub.io;
  access_log  /var/log/nginx/builder-access.log;
  error_log   /var/log/nginx/builder-error.log;

  ssl_certificate     /home/dokku/builder/tls/server.crt;
  ssl_certificate_key /home/dokku/builder/tls/server.key;
  ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;

  keepalive_timeout   70;


  location    / {

    gzip on;
    gzip_min_length  1100;
    gzip_buffers  4 32k;
    gzip_types    text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml  application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
    gzip_vary on;
    gzip_comp_level  6;

    proxy_pass  http://builder-5000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Request-Start $msec;
  }
  include /home/dokku/builder/nginx.conf.d/*.conf;
}

upstream builder-5000 {

  server 172.17.0.13:5000;
}

Environment details (AWS, VirtualBox, physical, etc.):

AWS

How was Dokku installed?:

As in the dokku manual.

How reproducible:

See curl/curl#1040 (comment)

Steps to Reproduce:

  1. Get a libcurl client with http2 support, e.g. compile with --with-nghttp2
  2. Run
curl -v https://builder.r-hub.io/api/check/validate_email \
 -H 'Content-Type: application/json' \
 -d '{"email":"somebody@gmail.com"}'

Actual Results:

Error.

Expected Results:

An HTTP response.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 28, 2016

Member

Can you make a pr here to fix the check?

Member

josegonzalez commented Sep 28, 2016

Can you make a pr here to fix the check?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 28, 2016

Member

Actually we need to probably disable both spdy and http2 if its between 1.9.5 and 1.11.0. Odd, I'm using 1.9.5 in prod and it works fine.

Member

josegonzalez commented Sep 28, 2016

Actually we need to probably disable both spdy and http2 if its between 1.9.5 and 1.11.0. Odd, I'm using 1.9.5 in prod and it works fine.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

@josegonzalez what is here? broken link....

maybe it is fine for GET requests, mine only fails for POST. And/or maybe 1.9.5 was fine, and they broke it later....

gaborcsardi commented Sep 28, 2016

@josegonzalez what is here? broken link....

maybe it is fine for GET requests, mine only fails for POST. And/or maybe 1.9.5 was fine, and they broke it later....

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 28, 2016

Member

Fixed the link.

Member

josegonzalez commented Sep 28, 2016

Fixed the link.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 28, 2016

So, what should it do then? Only allow 2.x and up and 1.11.x and up?

gaborcsardi commented Sep 28, 2016

So, what should it do then? Only allow 2.x and up and 1.11.x and up?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 28, 2016

Member

Right, but also disable spdy for the versions between, as otherwise they'll get spdy support, which will break as spdy isn't built in anymore.

Member

josegonzalez commented Sep 28, 2016

Right, but also disable spdy for the versions between, as otherwise they'll get spdy support, which will break as spdy isn't built in anymore.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 29, 2016

OK, I'll submit a PR today evening or tomorrow.

gaborcsardi commented Sep 29, 2016

OK, I'll submit a PR today evening or tomorrow.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Sep 29, 2016

Member

<3 thanks!

Member

josegonzalez commented Sep 29, 2016

<3 thanks!

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 30, 2016

It will take some more days, sorry. On a "brighter" note, we might even have problems with ngingx 1.11.3. Will investigate before submitting the PR here. :(

gaborcsardi commented Sep 30, 2016

It will take some more days, sorry. On a "brighter" note, we might even have problems with ngingx 1.11.3. Will investigate before submitting the PR here. :(

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Oct 1, 2016

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

gaborcsardi commented Oct 1, 2016

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Oct 1, 2016

Member

Did you have a specific error? "Not work well" isn't very helpful.

Member

josegonzalez commented Oct 1, 2016

Did you have a specific error? "Not work well" isn't very helpful.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Oct 1, 2016

I know, sorry, it was mostly a note for myself.

2016/10/01 06:45:41 [error] 9947#9947: *15600 connect() failed (113: No route to host) while connecting to upstream, client: 95.146.140.183, server: builder.r-hub.io, request: "POST /api/check/submit HTTP/1.1", upstream: "http://172.17.0.13:5000/api/check/submit", host: "builder.r-hub.io"

I think what happened was that I upgraded nginx, and everything was fine then, but then restarted some apps, and nginx started failing. Then I downgraded nginx, removed http2 support, and now everything is fine.

It could be unrelated, actually, so maybe 1.11.x is OK with dokku 0.7.x, I am not sure yet. I will need to pull up a test system to experiment, that's why it is taking time.

gaborcsardi commented Oct 1, 2016

I know, sorry, it was mostly a note for myself.

2016/10/01 06:45:41 [error] 9947#9947: *15600 connect() failed (113: No route to host) while connecting to upstream, client: 95.146.140.183, server: builder.r-hub.io, request: "POST /api/check/submit HTTP/1.1", upstream: "http://172.17.0.13:5000/api/check/submit", host: "builder.r-hub.io"

I think what happened was that I upgraded nginx, and everything was fine then, but then restarted some apps, and nginx started failing. Then I downgraded nginx, removed http2 support, and now everything is fine.

It could be unrelated, actually, so maybe 1.11.x is OK with dokku 0.7.x, I am not sure yet. I will need to pull up a test system to experiment, that's why it is taking time.

@oelmekki

This comment has been minimized.

Show comment
Hide comment
@oelmekki

oelmekki Oct 19, 2016

Hi guys, just to let you know that nginx-1.11.5 fixed it for me.

There's also something that may be of interest here:

  • I got REFUSED_STREAM error when a service using golang's net/http package tried to contact the http2 enabled dokku service
  • when I use curl on the same endpoint, nginx responds correctly
  • after using curl, the golang app correctly gets response from http2 service
  • after a while (max 1 minute), it starts failing again

This may be a clue that there's some http client configuration to do here to correctly connect to nginx.

EDIT: just to be clear, the http golang client was working without any problem on previous nginx version with spdy and it works just as fine with nginx-1.11.5 without any modification.

oelmekki commented Oct 19, 2016

Hi guys, just to let you know that nginx-1.11.5 fixed it for me.

There's also something that may be of interest here:

  • I got REFUSED_STREAM error when a service using golang's net/http package tried to contact the http2 enabled dokku service
  • when I use curl on the same endpoint, nginx responds correctly
  • after using curl, the golang app correctly gets response from http2 service
  • after a while (max 1 minute), it starts failing again

This may be a clue that there's some http client configuration to do here to correctly connect to nginx.

EDIT: just to be clear, the http golang client was working without any problem on previous nginx version with spdy and it works just as fine with nginx-1.11.5 without any modification.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Nov 22, 2016

Member

As the lowest known tested version with HTTP/2 support is Nginx 1.11.5, I'm going to drop support for http2 and spdy for anything above 1.9.5 and below 1.11.5.

I can't believe I'm doing feature detection in bash. Ugh.

Member

josegonzalez commented Nov 22, 2016

As the lowest known tested version with HTTP/2 support is Nginx 1.11.5, I'm going to drop support for http2 and spdy for anything above 1.9.5 and below 1.11.5.

I can't believe I'm doing feature detection in bash. Ugh.

josegonzalez added a commit that referenced this issue Nov 22, 2016

Detect nginx versions that support HTTP/2 well
Certain versions of nginx have bugs that prevent HTTP/2 from properly responding to all clients, thus causing applications to be unavailable.

Closes #2435
@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Nov 22, 2016

Member

Closing as a pull request is available.

Member

josegonzalez commented Nov 22, 2016

Closing as a pull request is available.

fruitl00p added a commit to fruitl00p/dokku that referenced this issue May 5, 2017

added SPDY/HTTP2 clearifications
in reference to #2435 and #2263 this should at least clearify which nginx versions result in SDPY/HTTP2 enabled TLS configurations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment