Unable to profixy a container started with docker-compose version 2

[JulienD]

Hi,

I wanted to implement the nginx-proxy to serve several nginx servers so I followed the example on the homepage and it works fine. However, thing start being more complex when I try to implement it with my real servers. In short the nginx-proxy is up but web servers are not proxified and I can only see a 503 status code from the proxy.

Nginx-proxy is catching events and is generating the Nginx default configuration file every time I start or stop a container.

dockergen.1 | 2016/07/08 13:38:53 Generated '/etc/nginx/conf.d/default.conf' from 1 containers
dockergen.1 | 2016/07/08 13:38:53 Running 'nginx -s reload'
dockergen.1 | 2016/07/08 13:38:53 Watching docker events
dockergen.1 | 2016/07/08 13:38:53 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'

But when I start on of my container I can see the following error message.

dockergen.1 | 2016/07/08 13:39:27 Received event start for container 69b904a872f0
dockergen.1 | 2016/07/08 13:39:28 Received event start for container 57e7ee6e4496
dockergen.1 | 2016/07/08 13:39:28 Generated '/etc/nginx/conf.d/default.conf' from 3 containers
dockergen.1 | 2016/07/08 13:39:28 Running 'nginx -s reload'
dockergen.1 | 2016/07/08 13:39:28 Error running notify command: nginx -s reload, exit status 1
dockergen.1 | 2016/07/08 13:39:28 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'

I had a look at the Nginx-proxy Default configuration and found that the generated file had no information in it about the web server I wanted to proxify.

upstream baz.bar.local {
}
server {
    server_name baz.bar.local;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://baz.bar.local;
    }
}

Has you can see there is nothing in the upstream section

upstream baz.bar.local {
}

I have done a lot of tests to understand what going on and finally found that problem only happens when I'm starting a container using docker-compose in version 2 because in other cases it works fine.

Here is a chunk of the Default Nginx file auto-generated

# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    listen 80;
    access_log /var/log/nginx/access.log vhost;
    return 503;
}
upstream bar.bar.local {
}
server {
    server_name bar.bar.local;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://bar.bar.local;
    }
}
upstream foo.bar.local {
                ## Can be connect with "bridge" network
            # whoamiversion1_whoami3_1
            server 172.17.0.4:8000;
}
server {
    server_name foo.bar.local;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://foo.bar.local;
    }
}
upstream red.bar.local {
                ## Can be connect with "bridge" network
            # whoami4
            server 172.17.0.3:8000;
}
server {
    server_name red.bar.local;
    listen 80 ;
    access_log /var/log/nginx/access.log vhost;
    location / {
        proxy_pass http://red.bar.local;
    }
}

Here is the workflow I have done to reproduce it

1. Start the nginx-proxy container

$ docker run -d --name nginx-proxy -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy

2. Start a container that needs to be proxified using docker command

$ docker run -e VIRTUAL_HOST=red.bar.local --name whoami4 jwilder/whoami

3. Start a container using docker-compose command version 2

The whoami_version-1/docker-compose.yml file

version: '2'
services:
  whoami1:
    image: jwilder/whoami
    environment:
      - VIRTUAL_HOST=bar.bar.local

$ docker-compose up

4. Start a container using docker-compose command version 1

whoami3:
  image: jwilder/whoami
  environment:
    - VIRTUAL_HOST=foo.bar.local

$ docker-compose up

I have been able to reproduce it several time on my local machine. I'm using the mac beta of Docker

$ docker --version
Docker version 1.12.0-rc3, build 91e29e8, experimental
$ docker-compose --version
docker-compose version 1.8.0-rc1, build 9bf6bc6

I think this issue a version a bit weird but I have not been able to find a better root cause.

Does anyone is able to reproduce ?

I have wrapped everything in a shell to help test easily. Unzip the archive and run startServers.sh

nginx_proxy_test.zip

[wader]

Hi, have look at this comment #478 (comment). What is probably happening is that docker-gen knows about the container but there is no common network. This is a known bug... generated config should probably just skip it totally or generate something valid in upstream {} indicating that there is no way of reaching the container(s).

[JulienD]

Thanks Wader for the info, I had a look at the issue queue but did not opened this topic. So it's a known bug.

[wader]

#438 is kind of this issue but the comments is a bit messy because of a similar issue happening at the same time.

[thesmartguild]

See the issue:

Problem with nginx.tmpl when using docker-compose based deployments. I believe that has the fix you need.

Todd

[dhrp]

FYI:
A way to get your v2 config file to work is to add:

network_mode: "bridge" to your docker-compose.yml file.

i.e. :

version: '2'
services:
  webserver:
    image: php:5.6-apache
    environment:
     - VIRTUAL_HOST=your.hostname.com
    network_mode: "bridge"

[brians-code]

I had the same issues, and solved it by creating a network called nginx-proxy:

docker create network nginx-proxy

Then I added the following to the bottom of my docker-compose file:

networks:
  default:
    external:
      name: nginx-proxy

Here is my complete docker-compose.yml:

version: '2'

services:

  prod:
    image: ${COMPOSE_PROJECT_NAME}-prod
    container_name: ${COMPOSE_PROJECT_NAME}-prod
    environment:
      - VIRTUAL_HOST=${COMPOSE_PROJECT_NAME}
    volumes:
      - ./prod/${COMPOSE_PROJECT_NAME}/web:/var/www/html/
      - ./prod/${COMPOSE_PROJECT_NAME}/data:/var/lib/mysql
    restart: always


  dev:
    image: ${COMPOSE_PROJECT_NAME}-dev
    container_name: ${COMPOSE_PROJECT_NAME}-dev
    environment:
      - VIRTUAL_HOST=${COMPOSE_PROJECT_NAME}.dev
    volumes:
      - ./dev/${COMPOSE_PROJECT_NAME}/web:/var/www/html
      - ./dev/${COMPOSE_PROJECT_NAME}/data:/var/lib/mysql


  nginx-proxy:
    image: jwilder/nginx-proxy
    container_name: nginx-proxy
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro


networks:
  default:
    external:
      name: nginx-proxy

[klokie]

That worked great for me - thanks @codekitchen-ws !

[LM1LC3N7]

I had the same problem, and I added to my docker-compose file this line :

network_mode: bridge

Whatever is the network_mode name, if you have created a new network. In my case, I use the default network.

I added this line in my gitlab docker-compose, it was the one who make crash the reverse proxy.

[schmunk42]

We're also seeing this issue, when not everything is configured 100% correctly. Our config is pretty similar to #552 (comment) (setup with overlay networking + compose v2)

As said before nginx-proxy should not create invalid configs, when the container is not reachable.

Moreover I haven't found a way to run compose v1 stacks with the above configuration. The upstreams are also empty, I had the public IP in the upstream before.

[schmunk42]

I added some more details about this issue and why it's happening here: #565 (comment)

[chao]

@briansrepo
Thanks a lot, it works for me, but the command of creating a new network should be docker network create proxy-nginx

[codepushr]

Is it possible that the network workaround is also needed with a plain Dockerfile, when not using compose and if so, how would I do that? I'm experiencing the same error without docker-compose.

[mintalicious]

Hey there, I ran into a similar problem, which I could not solve with the solution posted by @briansrepo in August 2016.

I want to run the nginx-proxy separately and then starting multiple compositions which all have another nginx running inside, mainly Laradocks.
I set up a new network, the proxy and all applications containers belong to, but I get 503 everytime.
The only way this is working is by setting network_mode: bridge in every container, so every container is in the bridge network.

It seems that the nginx-proxy has problems to include a new container in the config file when running in a custom network AND the new container is started from a separate composition.

# nginx-proxy/docker-compose.yml
version: '2'
services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    restart: always
    ports:
      - 80:80
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

networks:
  default:
    external:
      name: nginx-proxy

# app1/docker-compose.yml
version: '2'
services:
  nginx:
    image: myApp
    restart: always
    volumes:
      - ./htdocs:/var/www/html
    environment:
      - VIRTUAL_HOST=app1.dev

networks:
  default:
    external:
      name: nginx-proxy

This does not work. I created the network, started the proxy first and then the app.
Log of nginx-proxy shows
Generated '/etc/nginx/conf.d/default.conf' from 3 containers
and
Error running notify command: nginx -s reload, exit status 1

Whenn adding the above mentioned network_mode: bridge to both proxy and app container everything is fine.

Maybe someone has an idea about it.
Regards

[mintalicious]

The above behavior only appears on my test server. I do not have issues running this setup local!

Server:
Docker version 17.03.1-ce
Docker-compose version 1.12.0
OS Ubuntu 14.04.5 LTS
Kernel 3.13.0-101

Local:
Docker version 17.04.0-ce
Docker-compose version 1.11.2
OS Linux Mint 18
Kernel 4.10.0-20

[flocalhost]

Good day, i ran into another problem. When i start my first odoo:10.0 container with portainer stacks everything works fine:

version: "2"

services:
  web:
    restart: always
    image: odoo:10
    depends_on:
      - db
    ports:
      - "8069:8069"
    volumes:
      - odoo-web-data:/var/lib/odoo
      - ./config:/etc/odoo
      - ./addons:/mnt/extra-addons
    environment:
      - VIRTUAL_HOST=my.domain.com
      - VIRTUAL_PORT=8069
      - LETSENCRYPT_HOST=my.domain.com
      - LETSENCRYPT_EMAIL=admin@domain.com
    networks:
      - web
      - default
  db:
    restart: always
    image: postgres:9.6
    environment:
      - POSTGRES_PASSWORD=odoo
      - POSTGRES_USER=odoo
      - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - odoo-db-data:/var/lib/postgresql/data/pgdata
    networks:
      - web
      - default

volumes:
  odoo-web-data:
   driver: local
  odoo-db-data:
   driver: local

networks:
  web:
    external:
      driver: bridge
      name: mynet

also my nginx-proxy container is running in "mynet" network. But if i try to duplicate the docker-compose to this:

version: "2"

services:
  web:
    restart: always
    image: odoo:10
    depends_on:
      - db
    ports:
      - "8070:8069"
    volumes:
      - odoo-web-data:/var/lib/odoo
      - ./config:/etc/odoo
      - ./addons:/mnt/extra-addons
    environment:
      - VIRTUAL_HOST=my2.domain.com
      - VIRTUAL_PORT=8070
      - LETSENCRYPT_HOST=my2.domain.com
      - LETSENCRYPT_EMAIL=admin@domain.com
    networks:
      - web
      - default
  db:
    restart: always
    image: postgres:9.6
    environment:
      - POSTGRES_PASSWORD=odoo
      - POSTGRES_USER=odoo
      - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - odoo-db-data:/var/lib/postgresql/data/pgdata
    networks:
      - web
      - default

volumes:
  odoo-web-data:
   driver: local
  odoo-db-data:
   driver: local

networks:
  web:
    external:
      driver: bridge
      name: mynet

the nginx-proxy config said:

# my2.domain.com upstream my2.domain.com { ## Can be connected with "mynet" network # odoo3_web_1 server 172.18.0.9 down;

so its not possible to open up the container with my.2domain.com. its only reachable direct with http://IP:PORT

Any ideas?

[wader]

@flocalhost should VIRTUAL_PORT in the second config really be 8070 not 8069? that is assuming the odoo:10 image will losten on 8069

[flocalhost]

@flocalhost should VIRTUAL_PORT in the second config really be 8070 not 8069? that is assuming the odoo:10 image will losten on 8069

youre absolutely right. thanks a lot. worked hours on that problem

[tkw1536]

Resolved as per last comment.