`docker service logs` stops showing logs from containers on different nodes

[benturner]

Description
Running docker service logs foo on a swarm master where foo is a service with multiple replicas across different nodes eventually stops merging the logs from those other nodes. It seems to always work just fine right after the service is created.

Steps to reproduce the issue:

1. Create a service foo with replicas across multiple nodes
2. Run docker service logs --follow foo
3. Initially observe logs from multiple containers across different nodes
4. Go away and do something else for a while
5. Run docker service logs --follow foo
6. Observe old logs from multiple containers across different nodes but new logs only contain logs from the node on which you're running the command

Describe the results you received:
Logs from containers on the current node only

Describe the results you expected:
Logs from all containers on all nodes

Additional information you deem important (e.g. issue happens only occasionally):
Seems to work fine at first but then within some amount of time it stops working. I've tried with both json-file and journald log drivers.

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 7
 Running: 7
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 57
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: jbsbgj3on5coa7f996rle8bpk
 Is Manager: true
 ClusterID: 7uzbzxfjt8nf6p18wbzv8ek84
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.16.0.5
 Manager Addresses:
  172.16.0.5:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.636GiB
Name: swarmm-master-94917428-0
ID: NNL7:YHDL:5ALU:4ZXF:J3BL:VAIV:UI2T:TV5U:UGQL:UCQC:WWCP:TQDO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 149
 Goroutines: 310
 System Time: 2017-05-12T22:02:26.629917059Z
 EventsListeners: 7
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on azure using an acs-engine template (https://github.com/Azure/acs-engine). Currently just testing this so I'm using one manager and one worker node. The replicas for my service get split over both nodes.

[thaJeztah]

ping @dperny PTAL

[dperny]

Definitely a bug. @benturner about how long does it take for the logs to fall off? Also, since it's a test cluster, would it be possible for you to post daemon logs from about the time when you start the logs to the time when they fall off? If not, NBD, I will try to reproduce locally.

[benturner]

@dperny Yes I will try to grab some logs today. Thanks for looking into this.

[benturner]

Apologies, but I have tried and failed for a few days to get this to reproduce. I'll reopen if I can ever catch it. Thanks.

[thaJeztah]

Thanks! Keep us posted 👍

[yunghoy]

It should be reopend. This issue reproduced in 17.06.x-ce ~ 17.06.2-ce.

The first several times "docker service logs" command successfully gives logs.
After scaling (decrease and increase) containers more than 10 times, the command doesn't work anymore.

$ docker info

Containers: 49
Running: 42
Paused: 0
Stopped: 7
Images: 153
Server Version: 17.06.2-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 769
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: ciagobmjzqaup5typm21rcnzu
Is Manager: true
ClusterID: 86ntnntsfirsvrhwhhzjuy8tv
Managers: 2
Nodes: 3
Orchestration:
Task History Retention Limit: 1
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 3
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Root Rotation In Progress: false
Node Address: xxxxx
Manager Addresses:
xxxxx:2377
xxxxx:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.13.0-79-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 15.67GiB
Name: xxxxx
ID: XNBD:OS6A:OXQ6:RF5F:KW5S:VIRI:QT7M:SKTC:IK3J:WGVI:4H3K:TP6M
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

$ docker node ls

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0muzqhvzigqal85llqdhdxj64 xxxxx Ready Active Reachable
ciagobmjzqaup5typm21rcnzu * xxxxx Ready Active Leader
txoh28odun5kt42wuseo7qgdr xxxxx Ready Active

$ docker version

Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:00:33 2017
OS/Arch: linux/amd64

Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 19:59:26 2017
OS/Arch: linux/amd64
Experimental: false