Label ID is not assigned

[tgraf]

Bad: http://pastebin.com/QfmgVkUF
Good: http://pastebin.com/fFwpNpfY

[aanm]

@tgraf When this happens can you also provide the consul logs? docker logs cilium-consul

There's a 2016-06-14T02:47:38.182-07:00 node1 DEBU 03c lockPath > Creating lock for made and I was expecting to see a lockPath > Locked for right after that.

[tgraf]

vagrant@node1:~$ sudo docker logs cilium-consul
==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: '78436fe9bb2b'
Datacenter: 'dc1'
Server: true (bootstrap: true)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302)
Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
Atlas:

==> Log data will now stream in as it occurs:

2016/06/14 13:51:35 [INFO] serf: EventMemberJoin: 78436fe9bb2b 172.17.0.2
2016/06/14 13:51:35 [INFO] serf: EventMemberJoin: 78436fe9bb2b.dc1 172.17.0.2
2016/06/14 13:51:35 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state
2016/06/14 13:51:35 [INFO] consul: adding LAN server 78436fe9bb2b (Addr: 172.17.0.2:8300) (DC: dc1)
2016/06/14 13:51:35 [INFO] consul: adding WAN server 78436fe9bb2b.dc1 (Addr: 172.17.0.2:8300) (DC: dc1)
2016/06/14 13:51:35 [ERR] agent: failed to sync remote state: No cluster leader
2016/06/14 13:51:36 [ERR] http: Request GET /v1/kv/cilium-net/operational/LastUUID?index=63, error: No cluster leader from=172.17.0.1:54424
2016/06/14 13:51:36 [ERR] http: Request PUT /v1/session/create, error: No cluster leader from=172.17.0.1:54428
2016/06/14 13:51:36 [WARN] raft: Heartbeat timeout reached, starting election
2016/06/14 13:51:36 [INFO] raft: Node at 172.17.0.2:8300 [Candidate] entering Candidate state
2016/06/14 13:51:37 [INFO] raft: Election won. Tally: 1
2016/06/14 13:51:37 [INFO] raft: Node at 172.17.0.2:8300 [Leader] entering Leader state
2016/06/14 13:51:37 [INFO] consul: cluster leadership acquired
2016/06/14 13:51:37 [INFO] consul: New leader elected: 78436fe9bb2b
2016/06/14 13:51:37 [INFO] raft: Disabling EnableSingleNode (bootstrap)
2016/06/14 13:51:37 [INFO] consul: member '78436fe9bb2b' joined, marking health alive
2016/06/14 13:51:39 [INFO] agent: Synced service 'consul'

[tgraf]

I assume it's a race condition while consul did not finalize the leader election yet. We need to handle this gracefully.

[aanm]

Yes, I was expecting to see that if the timestamps were the same on the bad.txt and the cilium-consol log data.