Docker Swarm provider not reliable in v1.2.0-rc1 (Docker 1.13.0, traefik v1.2.0-rc1)

[SvenAbels]

What version of Traefik are you using (traefik version)?

v1.2.0-rc1

What is your environment & configuration (arguments, toml...)?

Docker Swarm Mode with Docker 1.13.0

What did you do?

I started traefik (dockerized verison) without changes. I used to use v1.1.2 and switched to v1.2.0-rc1. 1.1.2 works fine and discovers the services based on docker swarm labels. v1.2.0-rc1 does not do this when all services are deployed. the docker provider tab and its services are sometimes loaded correctly and sometimes not.

What did you expect to see?

the docker provider tab and a deployment of the docker swarm services

What did you see instead?

Looking at the Traefik log file, I can see:

time="2017-02-08T16:13:50Z" level=info msg="Starting provider *provider.Docker {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\"docker.localhost\",\"TLS\":null\"ExposedByDefault\":false,\"UseBindPortIP\":false,\"SwarmMode\":true}"

However, some lines below this I see:

time="2017-02-08T16:17:20Z" level=error msg="Near line 29 (last key parsed 'backends.backend-box.servers'): Key 'backends.backend-box.servers.server-box-1' has already been defined."
time="2017-02-08T16:17:35Z" level=error msg="Near line 127 (last key parsed 'backends.backend-rtdapp.servers'): Key 'backends.backend-rtdapp.servers.server-rtdapp-1' has already been defined."

This does not appear when restarting Traefik v1.1.2

[emilevauge]

@vdemeester any idea ?

[raarts]

I opened #1154 . My comment there about traefik 1.1.0 not having the problem may have been premature, because indeed restarting everything fixes it. I needed to restart the docker stack completely to get it to work again.

[christopherobin]

I've been debugging that for the past hour since I'm running in the same issue and what I found (at least for my case) is that the /tasks endpoint of docker will fail to return some of the data the provider is waiting for such as Slot or NetworkAttachements but traefik will still try to advertise it.

For the Slot missing part it seems that any service set to --mode=global will not get an actual slot ID, I'm not sure if it is expected behavior from Docker or a bug which would need to be fixed upstream, I guess a possible solution would be to fallback to NodeId if Slot is not defined?

For the NetworkAttachements it happened when I didn't specify any network, in my mind it should have defaulted to the ingress network that comes with swarm but docker might only try to attach it if the containers tries to access it? Anyway the fix for that one is to specify a network manually, though I think traefik should not try to advertise it if there are no networks attached to the task.

[allamand]

I reproduce this error with a service set on a node and running, then I delete that node. Docker Swarm re-spawn my container with the same name on another Node,

ID            NAME                         IMAGE                      NODE          DESIRED STATE  CURRENT STATE           ERROR  PORTS
xe7lt5q08yw7  visualizer_visualizer.1      manomarks/visualizer:beta  dfy-master01  Running        Running 5 minutes ago          
5ux97hzmi1fg   \_ visualizer_visualizer.1  manomarks/visualizer:beta  dfy-master04  Shutdown       Running 12 minutes ago  

but then Traefik don't update correctly it's configuration with the error message

time="2017-02-16T12:40:32Z" level=error msg="Near line 57 (last key parsed 'backends.backend-visualizer.servers'): Key 'backends.backend-visualizer.servers.server-visualizer_visualizer-1' has already been defined." 

[SvenAbels]

Hi guys,

just a note because I saw that #1167 is mentioned here: In our servers, we only got non-global tasks (with replica 1) so the problem above seems to be independent of global tasks

[christopherobin]

@ascora It is probably due to the 2nd thing I fixed in my PR which are dead tasks not being filtered properly (as shown by @allamand), dead tasks still have a slot ID (usually the one they occupied when they died) so they would end up having the same backend name as running tasks (<service_name>.<slot>) and would cause the name collision you see in your logs.

[emilevauge]

Fixed by #1167

[bitsofinfo]

I am still seeing this error on 1.3.3 with replicated mode, it only shows up when I do a docker service scale my-service=[n]