server ip is marked as down when using different internal networks

[gtaspider]

Got a docker-compose file which will have multiple internal docker networks. The proxy seems to find the correct ip and set it to the correct upstream but the server is marked as down, eventhough the commet says nginx was able to connect to the right network:

default.conf:

#[...]
# cloud.website.com
upstream cloud.website.com {
				# Cannot connect to network of this container
				server 127.0.0.1 down;
				## Can be connected with "test_net-dashboard" network
		# test_nextcloud_1
			server 172.31.0.4 down;
}
server {
	server_name cloud.website.com;
	listen 80 ;
	access_log /var/log/nginx/access.log vhost;
	location / {
		proxy_pass http://cloud.website.com;
	}
}

When I remove the down, safe it and then reload nginx with docker exec -it test_proxy_1 nginx -s reload it works like a charm. But if a container starts/stops the file will be rewritten (this is why I want to use this tool..)

I created a simple docker-compose file, to show the problem:

version: '3'

services:
  db:
    image: postgres:alpine
    restart: always
    volumes:
      - db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=
      - POSTGRES_DB=nextcloud
      - POSTGRES_USER=usr
    networks:
      - net-dashboard
  nextcloud:  
    image: nextcloud:apache
    restart: always
    volumes:
      - nextcloud:/var/www/html
    environment:
      - VIRTUAL_HOST=cloud.website.com
      - POSTGRES_HOST=db
      - MM_USERNAME=usr
      - MM_PASSWORD=
      - MM_DBNAME=nextcloud
    depends_on:
      - db
    networks:
     - net-dashboard
  #NGINX Proxy
  proxy:
    image: jwilder/nginx-proxy:alpine
    restart: always
    ports:
      - 80:80
    volumes:
#      - certs:/etc/nginx/certs:ro
      - vhost.d:/etc/nginx/vhost.d
      - conf.d:/etc/nginx/conf.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - net-dashboard
      - default
 

volumes:
  db:
  nextcloud:
#  certs:
  vhost.d:
  conf.d:
  html:


networks:
  net-dashboard:
    internal: true

This seems to be a bug, so I posted it here.
Thanks in advance,
Spider

[phhutter]

Hi Spider,

same issue here.
tested with jwilder 0.6.0 0.7.0 and redhat 7.5 docker 1.13

I've noticed, that when you startup a container with an internal network only, the ports wont get displayed in a simple docker ps. Is it possible that the proxy can't fetch the port informations as well ?

[root@in001 opt]# docker network create int_network --internal
20985d85d58aef2880642422fa9e5649babada8d8d556327dfc64f693f1f772e

[root@in001 opt]# docker run -d --network int_network -p 80:80 nginx
544847612ffe5f3582c449ada154bb4774598d14fd9f9b13433a04c9fb7cab64

[root@in001 opt]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
544847612ffe        nginx               "nginx -g 'daemon ..."   3 seconds ago       Up 2 seconds                               awesome_lamport

When i run a docker inspect i can see the ExposedPorts, even if its an internal network. But the "Ports"-section is empty.

....
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.13.7-1~stretch",
                "NJS_VERSION=1.13.7.0.1.15-1~stretch"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "ArgsEscaped": true,
            "Image": "nginx",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "43831a9bc2278a82023019733015148a5bca2d9f20bd1fd63a175203c68d5fad",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
...

would appreciate if anyone can verify that.
regards,
phhutter

[phhutter]

i'm digging ...

just need to know how the values get filled into the template.

{{ $CurrentContainer := where $ "ID" .Docker.CurrentContainerID | first }}

{{ define "upstream" }}
        {{ if .Address }}
                {{/* If we got the containers from swarm and this container's port is published to host, use host IP:PORT */}}
                {{ if and .Container.Node.ID .Address.HostPort }}
                        # {{ .Container.Node.Name }}/{{ .Container.Name }}
                        server {{ .Container.Node.Address.IP }}:{{ .Address.HostPort }};
                {{/* If there is no swarm node or the port is not published on host, use container's IP:PORT */}}
                {{ else if .Network }}
                        # {{ .Container.Name }}
                        server {{ .Network.IP }}:{{ .Address.Port }};
                {{ end }}
        {{ else if .Network }}
                # {{ .Container.Name }}
                {{ if .Network.IP }}
                        server {{ .Network.IP }} down;
                {{ else }}
                        server 127.0.0.1 down;
                {{ end }}
        {{ end }}

{{ end }}

[phhutter]

think the problem would be an empty .Address variable.

#https://github.com/jwilder/docker-gen/blob/a4f4f0167148a5ae52ce063474d850ef8b78f93c/generator.go

		for k, v := range container.NetworkSettings.Ports {
			address := Address{
				IP:           container.NetworkSettings.IPAddress,
				IP6LinkLocal: container.NetworkSettings.LinkLocalIPv6Address,
				IP6Global:    container.NetworkSettings.GlobalIPv6Address,
				Port:         k.Port(),
				Proto:        k.Proto(),
}

but when i inspect the containers, i will get an empty array from the internal network container

external:

[root@in001 ~]# docker inspect --format=" {{ .NetworkSettings.Ports }} " d99315310f36
 map[80/tcp:[{0.0.0.0 80}]]

internal:

[root@in001 ~]# docker inspect --format=" {{ .NetworkSettings.Ports }} " 70c4c3b1d313
 map[]

[gtaspider]

I can verify your posts. So there is actually no workaround, right?
Thanks for your support!

[meron1122]

If you don't need separate subnet, but you have separate docker-compose, simple workaround is setting up default network to join nginx network.

Add this of the bottom of your app docker-compose file:

networks:
  default:
    external:
      name: webproxy

[gtaspider]

Thanks for your response but unfortunately I need an seperate internal subnet for security purposes ...

[lunedis]

ran into the same issue, any plan on this getting fixed? or is it working as intended / wontfix?

[phhutter]

At the moment theres a workaround, just add a 2nd (not internal) network.

But as gtaspider said, for security reason the generator.go needs a fix. Im not sure if its possible and why the guys took the .Address parameter instead of another one who will show the exposed ports.

Ps. Think it would fix following issue as well #1066.

[webdevotion]

Thanks @meron1122 for the pointer.

I can confirm that removing all internal networks from my app's docker-compose.yml fixed it for me. My default.conf had similar lines with server 127.0.0.1 down; under the upstream entry.

After removing all internal networks and declaring the default, external network everything started working as expected. Mixing in the webproxy network together with other internal networks did not work for me.

For future Googlers.

Put the snippet from @meron1122's comment at the bottom of your docker compose file and remove all other entries of networks in each of the container's configuration. Please understand that there might be security implications as pointed out by @gtaspider.

Important notice

Don't forget to create the external network webproxy. That network will be available for all your dockerized environments. You can do so by running:

$ docker network create webproxy

This might help

You might want to stop all your services ( nginx-proxy and your application's stack ) before restarting them one at a time. First the nginx-proxy stack, then your application(s).

I am only using the webproxy network

No other networks are used anywhere in my docker-compose.yml file. I will probably try to get things working again with multiple, ( internal, ) networks to split up frontend, backend, ... and so on. But his works for now. As you can read in the comments above, it seems to be quite a challenge to find a permanent fix.

[zedtux]

I'm on the same boat :(

[muesli]

I'm experiencing the same issue for any container that exposes two ports (e.g. gitea, one for SSH & one for HTTP), even though it is in the same network as the nginx-proxy. I'm not sure if it's really related, but since a few people here already debugged the surrounding code, I'm wondering if there's any advice for this scenario?

[frederikbosch]

@muesli That can be solved by using PR #1157. I don’t know if it also solves the problem mentioned by @gtaspider.

[phhutter]

We should give a shot. It seems to solve the issue (just reviewed the changes), but havnt tested it now.

[willtho89]

I can use multiple internal networks with nginx-gen and an older template. Here is the diff between the two:

diff working.tmpl nginx.tmpl
21a22
>
132c133
<                 ## Can be connect with "{{ $containerNetwork.Name }}" network
---
>                 ## Can be connected with "{{ $containerNetwork.Name }}" network
143a145,147
>             {{ else }}
>                 # Cannot connect to network of this container
>                 server 127.0.0.1 down;
159,161d162
< {{/* Get the NETWORK_ACCESS defined by containers w/ the same vhost, falling back to "external" */}}
< {{ $network_tag := or (first (groupByKeys $containers "Env.NETWORK_ACCESS")) "external" }}
<

This was committed in #1106 to fix #1105.