Doesn't work with multiple networks in swarm mode

[Vanuan]

Do you want to request a feature or report a bug?

Bug

What did you do?

docker network create --driver overlay traefik-net
docker network create --driver overlay app-net

Compose file for traefik:

version: '3.1'

services:
  proxy:
    image: traefik:1.6.4-alpine
    networks:
     - traefik-net
    command: --web --docker --docker.domain=$TRAEFIK_DOMAIN docker.swarmmode=true --loglevel=DEBUG --docker.watch
    ports:
      - "30000:80"
      - "30443:443"
      - "30080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /dev/null:/traefik.toml

networks:
  traefik-net:
    external: true

Deploy traefik:

TRAEFIK_DOMAIN=traefik docker stack deploy -c deploy.yml traefik

Deploy app:

docker service create     --name whoami0     --container-label traefik.port=80 --container-label traefik.frontend.rule=Host:whoami0.traefik  --network traefik-net  --network app-net   emilevauge/whoami

What did you expect to see?

I expected http://whoami0.traefik:30000 to work

What did you see instead?

Gateway Timeout

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.6.4
Codename:     tetedemoine
Go version:   go1.10.3
Built:        2018-06-15_03:12:50PM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Docker version 18.05.0-ce, build f150324

--web --docker --docker.domain=$TRAEFIK_DOMAIN docker.swarmmode=true --loglevel=DEBUG --docker.watch

If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)

time="2018-06-27T15:08:52Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2018-06-27T15:08:52Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://10.0.1.4:80" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2018-06-27T15:08:52Z" level=debug msg="vulcand/oxy/forward: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.1.4:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2018-06-27T15:08:52Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.1.4:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2018-06-27T15:09:22Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://10.0.1.4:80, code: 504, Length: 15, duration: 30.000565877s"
time="2018-06-27T15:09:22Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.1.4:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2018-06-27T15:09:22Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.0.1.4:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2018-06-27T15:09:22Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en,en-US;q=0.9,ru;q=0.8,ru-RU;q=0.7,uk;q=0.6\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.7 Safari/537.36\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami0.traefik:30000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.255.0.2:40271\",\"RequestURI\":\"/\",\"TLS\":null}"

[Vanuan]

docker network inspect app-net

        "Containers": {
            "139a7c1ec1b367615bfe0267b747747bd9286e0b2769879110835023cbe9356f": {
                "Name": "whoami0.1.aqpfd2m6s5p01lrvu1rm7ehxc",
                "EndpointID": "a5adde90affa287441bdddc622870542ffc5524e4ca39f17a777979a143c11f4",
                "MacAddress": "02:42:0a:00:01:04",
                "IPv4Address": "10.0.1.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {

docker network inspect traefik-net

        "Containers": {
            "074a8ff6b6c3e79af5791d66603a7c8a10da1d22c871f8449cfe811ce426e0e7": {
                "Name": "traefik_proxy.1.f6gnifwrn5f3oqc8p30d7mvo8",
                "EndpointID": "02f0a140e6d6a23a5c3c6d1e436bae82f8ac4e4face85f3672f0909a3898b9b7",
                "MacAddress": "02:42:0a:00:02:04",
                "IPv4Address": "10.0.2.4/24",
                "IPv6Address": ""
            },
            "1a5278a8437d7f851426d31c25f103b255c3359c0431e28a6be493d88a745f16": {
                "Name": "whoami0.1.x3j78ml1mmtgj86fce9j4vsat",
                "EndpointID": "0f97064fbd677e07cfa92a77fd0674daf45143033083a8b2272befb453bd22a7",
                "MacAddress": "02:42:0a:00:02:06",
                "IPv4Address": "10.0.2.6/24",
                "IPv6Address": ""
            }
        },

[Vanuan]

So traefik chooses a wrong ip while connecting to whoami0. Let's try to use this "traefik.docker.network" label.

docker service rm whoami0
docker service create     --name whoami0     --container-label traefik.port=80 --container-label traefik.frontend.rule=Host:whoami0.traefik --container-label traefik.docker.network=traefik-net --network traefik-net  --network app-net   emilevauge/whoami

And it works... I swear it didn't work before...

[traefiker]

Hi! I'm Træfiker 🤖 the bot in charge of tidying up the issues.

Thanks again for your interest in Træfik!

I'm glad to close the issue because I think that the question has been answered (Yes, I'm quite an advanced AI 😄)

Join our Slack workspace for more community #support.

[Vanuan]

Ok, the issue I was having is that I missed dashes in docker.swarmmode=true:

--web --docker --docker.domain=$TRAEFIK_DOMAIN docker.swarmmode=true --loglevel=DEBUG --docker.watch

So it only worked when it was on the same machine :)
Hope nobody else made such a silly mistake.

[Blackfaded]

Had the same issue, thanks for the docker.network label hint!