Do not allow http2 for nginx 1.10.x

[gaborcsardi]

Description of problem:

dokku turns on http2 for nginx 1.10.x and above. But this version of nginx has a broken http2 implementation, and http2 aware clients fail to connect to it. Here is an example case: curl/curl#1040

nginx 1.11.x seems to be fine.

Output of the following commands

• uname -a:

Linux ip-172-31-16-56 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• free -m:

             total       used       free     shared    buffers     cached
Mem:          3952       3260        691          6        318       1369
-/+ buffers/cache:       1571       2380
Swap:            0          0          0

• docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:22:43 2016
 OS/Arch:      linux/amd64

• docker -D info:

Containers: 28
 Running: 12
 Paused: 0
 Stopped: 16
Images: 151
Server Version: 1.12.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 316
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-77-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: ip-172-31-16-56
ID: 2RSZ:VYWR:Z577:KLV4:DVHJ:AHSQ:656I:FBOG:XSHB:FOJS:PXUD:TKZU
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

• sigil -v:

0.4.0

• docker run -ti gliderlabs/herokuish:latest herokuish version:

herokuish: 0.3.18
buildpacks:
  heroku-buildpack-multi     v1.0.0
  heroku-buildpack-ruby      v146
  heroku-buildpack-nodejs    v91
  heroku-buildpack-clojure   v75
  heroku-buildpack-python    v81
  heroku-buildpack-java      v44
  heroku-buildpack-gradle    v17
  heroku-buildpack-grails    v21
  heroku-buildpack-scala     v72
  heroku-buildpack-play      v26
  heroku-buildpack-php       v109
  heroku-buildpack-go        v44
  heroku-buildpack-erlang    fa17af9
  buildpack-nginx            v5
  emberjs.tgz                latest

• dokku version:

0.7.1

• dokku plugin:

plugn: dev
  00_dokku-standard    0.7.1 enabled    dokku core standard plugin
  20_events            0.7.1 enabled    dokku core events logging plugin
  apps                 0.7.1 enabled    dokku core apps plugin
  build-env            0.7.1 enabled    dokku core build-env plugin
  certs                0.7.1 enabled    dokku core certificate management plugin
  checks               0.7.1 enabled    dokku core checks plugin
  common               0.7.1 enabled    dokku core common plugin
  config               0.7.1 enabled    dokku core config plugin
  docker-options       0.7.1 enabled    dokku core docker-options plugin
  domains              0.7.1 enabled    dokku core domains plugin
  enter                0.7.1 enabled    dokku core enter plugin
  git                  0.7.1 enabled    dokku core git plugin
  jenkins              1.0.0 enabled    dokku jenkins service plugin with R installed
  logs                 0.7.1 enabled    dokku core logs plugin
  named-containers     0.7.1 enabled    dokku core named containers plugin
  nginx-vhosts         0.7.1 enabled    dokku core nginx-vhosts plugin
  plugin               0.7.1 enabled    dokku core plugin plugin
  proxy                0.7.1 enabled    dokku core proxy plugin
  ps                   0.7.1 enabled    dokku core ps plugin
  rabbitmq             1.0.0 enabled    dokku rabbitmq service plugin
  redirect             0.4.0 enabled    Simple redirects for apps
  redis                1.0.0 enabled    dokku redis service plugin
  repo                 0.7.1 enabled    dokku core repo plugin
  shell                0.7.1 enabled    dokku core shell plugin
  ssh-keys             0.7.1 enabled    dokku core ssh-keys plugin
  storage              0.7.1 enabled    dokku core storage plugin
  tags                 0.7.1 enabled    dokku core tags plugin
  tar                  0.7.1 enabled    dokku core tar plugin

• cat /home/dokku/<app>/nginx.conf (if applicable):

server {
  listen      [::]:80;
  listen      80;
  server_name r-hub.io;
  access_log  off;
  return 301  $scheme://log.r-hub.io$request_uri;
}

server {
  listen      [::]:443 ssl http2;
  listen      443 ssl http2;
  server_name r-hub.io;
  access_log  off;

  ssl_certificate     /home/dokku/builder/tls/server.crt;
  ssl_certificate_key /home/dokku/builder/tls/server.key;

  return 301  $scheme://log.r-hub.io$request_uri;
}


server {
  listen      [::]:80;
  listen      80;
  server_name builder.r-hub.io;
  access_log  /var/log/nginx/builder-access.log;
  error_log   /var/log/nginx/builder-error.log;

  return 301 https://$host:443$request_uri;

}

server {
  listen      [::]:443 ssl http2;
  listen      443 ssl http2;
  server_name builder.r-hub.io;
  server_name builder.r-hub.io;
  access_log  /var/log/nginx/builder-access.log;
  error_log   /var/log/nginx/builder-error.log;

  ssl_certificate     /home/dokku/builder/tls/server.crt;
  ssl_certificate_key /home/dokku/builder/tls/server.key;
  ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;

  keepalive_timeout   70;


  location    / {

    gzip on;
    gzip_min_length  1100;
    gzip_buffers  4 32k;
    gzip_types    text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml  application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
    gzip_vary on;
    gzip_comp_level  6;

    proxy_pass  http://builder-5000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Request-Start $msec;
  }
  include /home/dokku/builder/nginx.conf.d/*.conf;
}

upstream builder-5000 {

  server 172.17.0.13:5000;
}

• Link to the exact repository being deployed (if possible/applicable):
  https://github.com/r-hub/rhub-frontend

Environment details (AWS, VirtualBox, physical, etc.):

AWS

How was Dokku installed?:

As in the dokku manual.

How reproducible:

See curl/curl#1040 (comment)

Steps to Reproduce:

1. Get a libcurl client with http2 support, e.g. compile with --with-nghttp2
2. Run

curl -v https://builder.r-hub.io/api/check/validate_email \
 -H 'Content-Type: application/json' \
 -d '{"email":"somebody@gmail.com"}'

Actual Results:

Error.

Expected Results:

An HTTP response.

[josegonzalez]

Can you make a pr here to fix the check?

[josegonzalez]

Actually we need to probably disable both spdy and http2 if its between 1.9.5 and 1.11.0. Odd, I'm using 1.9.5 in prod and it works fine.

[gaborcsardi]

@josegonzalez what is here? broken link....

maybe it is fine for GET requests, mine only fails for POST. And/or maybe 1.9.5 was fine, and they broke it later....

[josegonzalez]

Fixed the link.

[gaborcsardi]

So, what should it do then? Only allow 2.x and up and 1.11.x and up?

[josegonzalez]

Right, but also disable spdy for the versions between, as otherwise they'll get spdy support, which will break as spdy isn't built in anymore.

[gaborcsardi]

OK, I'll submit a PR today evening or tomorrow.

[josegonzalez]

<3 thanks!

[gaborcsardi]

It will take some more days, sorry. On a "brighter" note, we might even have problems with ngingx 1.11.3. Will investigate before submitting the PR here. :(

[gaborcsardi]

For the record 1.11.3 didn't work well with dokku, not sure why, but I had to downgrade quickly and disable http2.

[josegonzalez]

Did you have a specific error? "Not work well" isn't very helpful.

[gaborcsardi]

I know, sorry, it was mostly a note for myself.

2016/10/01 06:45:41 [error] 9947#9947: *15600 connect() failed (113: No route to host) while connecting to upstream, client: 95.146.140.183, server: builder.r-hub.io, request: "POST /api/check/submit HTTP/1.1", upstream: "http://172.17.0.13:5000/api/check/submit", host: "builder.r-hub.io"

I think what happened was that I upgraded nginx, and everything was fine then, but then restarted some apps, and nginx started failing. Then I downgraded nginx, removed http2 support, and now everything is fine.

It could be unrelated, actually, so maybe 1.11.x is OK with dokku 0.7.x, I am not sure yet. I will need to pull up a test system to experiment, that's why it is taking time.

[oelmekki]

Hi guys, just to let you know that nginx-1.11.5 fixed it for me.

There's also something that may be of interest here:

• I got REFUSED_STREAM error when a service using golang's net/http package tried to contact the http2 enabled dokku service
• when I use curl on the same endpoint, nginx responds correctly
• after using curl, the golang app correctly gets response from http2 service
• after a while (max 1 minute), it starts failing again

This may be a clue that there's some http client configuration to do here to correctly connect to nginx.

EDIT: just to be clear, the http golang client was working without any problem on previous nginx version with spdy and it works just as fine with nginx-1.11.5 without any modification.