Help: 502 Bad Gateway

[ealves-pt]

Hey everybody, I'm struggling here with an issue that I didn't manage to solve.

Basically my host runs Ubuntu 14.04. I have followed this and setup /etc/default/docker with DOCKER_OPTS="--iptables=false" and /etc/default/ufw DEFAULT_FORWARD_POLICY="ACCEPT".

I've crated a custom docker network with:

docker network create nginx-proxy

I've set on top of /etc/ufw/before.rules (I'm actually not sure if this step was applied successfully):

# nat Table rules
*nat
:POSTROUTING ACCEPT [0:0]

# Forward traffic from eth1 through eth0.
-A POSTROUTING -s 172.19.0.0/16 -o eth0 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT

I have the two following docker-composer.yml:

nginx/docker-compose.yml:

version: "2"
services:
  nginx:
    image: nginx
    container_name: nginx
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "./conf.d:/etc/nginx/conf.d"
      - "./vhost.d:/etc/nginx/vhost.d"
      - "./html:/usr/share/nginx/html"
    networks:
      - proxy-tier

  docker-gen:
    image: jwilder/docker-gen
    container_name: docker-gen
    restart: always
    volumes_from:
      - nginx
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
      - "./nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro"
      - "./certs:/etc/nginx/certs:ro"
    links:
      - nginx
    networks:
      - proxy-tier
    entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

networks:
  proxy-tier:
    external:
      name: nginx-proxy

rancher/docker-compose.yml:

version: "2"
services:
  master:
    image: rancher/server:latest
    volumes:
      - /data-docker/rancher-server:/var/lib/mysql
    restart: always
    environment:
      - VIRTUAL_HOST=example.com
      - VIRTUAL_PORT=8080
    networks:
      - proxy-tier

networks:
  proxy-tier:
    external:
      name: nginx-proxy

My nginx/conf.d/default looks like this:

upstream example.com {
                ## Can be connect with "nginx-proxy" network
            # rancherserver_master_1
            server 172.19.0.2:8080;
}
server {
    server_name example.com;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://example.com;
    }
}

And I also get the following nginx errors:

docker-gen | 2016/06/25 23:09:54 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
docker-gen | 2016/06/25 23:09:54 Watching docker events
docker-gen | 2016/06/25 23:09:54 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
nginx      | 2016/06/25 23:09:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "http://172.19.0.2:8080/", host: "example.com"
nginx      | example.com 172.19.0.1 - - [25/Jun/2016:23:09:57 +0000] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
nginx      | 2016/06/25 23:09:57 [error] 5#5: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.19.0.2:8080/favicon.ico", host: "example.com", referrer: "http://example.com/"
nginx      | example.com 172.19.0.1 - - [25/Jun/2016:23:09:57 +0000] "GET /favicon.ico HTTP/1.1" 502 575 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"

But I'm still not able to reach rancher from the outside. Any hints on how to tackle this? I think I went through all the open and closed issues here but I wasn't able to get it working.

[wader]

Hmm i would start out by docker exec into nginx container and see what it actually can reach, might have to install nc, curl etc. Maybe poke around and dump iptables, have look at https://stackoverflow.com/questions/31265993/docker-networking-namespace-not-visible-in-ip-netns-list how to work with network namespaces and docker.
Also the hardcoded 172.19.0.0/16 looks a bit fishy, docker dynamically allocates networks... so things might work for a while but stop after container recreate etc.
Hope that helps a bit. I might have a look at this later today.

[ealves-pt]

@wader basically I'm out of ideas. No luck messing around with iptables and trying to curl form the nginx container I was not able to get anything from the rancher container that is on the same network.

I'm completely annoyed with this because there is no easy way to make docker and ufw work together and that makes it so hard to setup nginx-proxy properly.

[wader]

@ealves-pt Ok, sorry I haven't tried anything yet. But i would suggest experiment without involving nginx-proxy just to have less moving parts. What is the end goal, just to make docker and ufw happy together?

[ealves-pt]

@wader yeah the first goal is to have docker and ufw working together. I'll make sure I get that one right and then I will add nginx-proxy to the stack.

[rossPatton]

Struggling with this. New to docker, nginx, etc. Read all the comments here and on the other PRs, and still confused.

The app works fine when I use the port (if I export it via ports in my docker-compose, although this just means the nginx container isn't working i'm guessing), and gives the nginx 502 error page when I try to just use the alias

I followed your tutorial here exactly: https://medium.com/@francoisromain/set-a-local-web-development-environment-with-custom-urls-and-https-3fbe91d2eaf0

I tried looking at the access log, and the error log, in the var/log/nginx folder and both were empty.

I don't know much about nginx, but a few things stand out to me?

• the ssl key and crt files are correct
• i don't know where this 172.28.0.4 ip comes from in the upstream consensus.local block
• resolver being 127.0.0.11 seems weird to me but when i changed it to 127.0.0.1 it didn't help
• when i try to access 172.28.0.4 or 172.28.0.4:3001 my connection just times out
• when i try to access consensus.local i DO get redirected to https, and then I get the 502
• i am not running apache and my local nginx file is the default

My hosts file:

127.0.0.1 consensus.local

My nginx compose file:

version: "3.1"

services:
  nginx-proxy:
    image: jwilder/nginx-proxy:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./certs:/etc/nginx/certs
      - /var/run/docker.sock:/tmp/docker.sock:ro
    restart: unless-stopped

networks:
  default:
    external:
      name: nginx-proxy

my main compose file:

# local development
version: '3'
services:
  dev:
    container_name: dev
    command: "npx concurrently \"webpack --watch --config webpack/\" \"npx pm2-dev pm2.config.js --no-daemon\""
    depends_on:
      - db
    env_file:
      - .env
    environment:
      - DB=development
      - DEBUG=false
      - NODE_ENV=development
      - VIRTUAL_HOST=consensus.local
      - VIRTUAL_PORT=3001
    expose:
      - 3001
      - 9229
    image: node:12.3.0-alpine
    # ports:
    #   - "3001:3001"
    #   - "9229:9229"
    working_dir: /app
    volumes:
      - nodemodules:/app/node_modules
      - .:/app

  db:
    container_name: db
    image: postgres:11.2-alpine
    ports:
      - "54320:5432"
    volumes:
      - "db_data:/var/lib/postgresql/data"

  redis:
    container_name: redis
    image: redis:alpine
    sysctls:
      # fixes warning when using redis with the barebones alpine image
      net.core.somaxconn: '511'

# connect our localhost to the nginx reverse proxy
networks:
  default:
    external:
      name: nginx-proxy

# ties our local directory and db to the running docker container
volumes:
  db_data:
    external: true
  nodemodules:
    external: true

Docker ps showing all the containers running:

nginx default.conf in the running container:

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}

# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header
map $scheme $proxy_x_forwarded_ssl {
  default off;
  https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
                ssl_protocols TLSv1.2 TLSv1.3;
                ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA->
                ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        listen 80;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
# consensus.local
upstream consensus.local {
                                ## Can be connected with "nginx-proxy" network
                        # dev
                        server 3001;
}
server {
        server_name consensus.local;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        return 301 https://$host$request_uri;
}
server {
        server_name consensus.local;
        listen 443 ssl http2 ;
        access_log /var/log/nginx/access.log vhost;
        ssl_session_timeout 5m;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;
        ssl_certificate /etc/nginx/certs/consensus.local.crt;
        ssl_certificate_key /etc/nginx/certs/consensus.local.key;
        add_header Strict-Transport-Security "max-age=31536000" always;
        location / {
                proxy_pass http://consensus.local;
        }
}

[maxlyaptsev]

I had had the same problem, but it looks like jwilder/nginx-proxy creates wrong nginx.conf file.
If you change

location / {
                proxy_pass http://consensus.local;
 }

to

location / {
                proxy_pass http://dev;
        }

it will work fine.