Repository navigation
Description
Description
Running docker service logs foo on a swarm master where foo is a service with multiple replicas across different nodes eventually stops merging the logs from those other nodes. It seems to always work just fine right after the service is created.
Steps to reproduce the issue:
- Create a service
foowith replicas across multiple nodes - Run
docker service logs --follow foo - Initially observe logs from multiple containers across different nodes
- Go away and do something else for a while
- Run
docker service logs --follow foo - Observe old logs from multiple containers across different nodes but new logs only contain logs from the node on which you're running the command
Describe the results you received:
Logs from containers on the current node only
Describe the results you expected:
Logs from all containers on all nodes
Additional information you deem important (e.g. issue happens only occasionally):
Seems to work fine at first but then within some amount of time it stops working. I've tried with both json-file and journald log drivers.
Output of docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: true
Output of docker info:
Containers: 7
Running: 7
Paused: 0
Stopped: 0
Images: 6
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 57
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: active
NodeID: jbsbgj3on5coa7f996rle8bpk
Is Manager: true
ClusterID: 7uzbzxfjt8nf6p18wbzv8ek84
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 172.16.0.5
Manager Addresses:
172.16.0.5:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.636GiB
Name: swarmm-master-94917428-0
ID: NNL7:YHDL:5ALU:4ZXF:J3BL:VAIV:UI2T:TV5U:UGQL:UCQC:WWCP:TQDO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 149
Goroutines: 310
System Time: 2017-05-12T22:02:26.629917059Z
EventsListeners: 7
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
Running on azure using an acs-engine template (https://github.com/Azure/acs-engine). Currently just testing this so I'm using one manager and one worker node. The replicas for my service get split over both nodes.
Activity
thaJeztah commentedon May 15, 2017
ping @dperny PTAL
dperny commentedon May 15, 2017
Definitely a bug. @benturner about how long does it take for the logs to fall off? Also, since it's a test cluster, would it be possible for you to post daemon logs from about the time when you start the logs to the time when they fall off? If not, NBD, I will try to reproduce locally.
benturner commentedon May 15, 2017
@dperny Yes I will try to grab some logs today. Thanks for looking into this.
benturner commentedon May 18, 2017
Apologies, but I have tried and failed for a few days to get this to reproduce. I'll reopen if I can ever catch it. Thanks.
thaJeztah commentedon May 18, 2017
Thanks! Keep us posted 👍
yunghoy commentedon Sep 19, 2017
It should be reopend. This issue reproduced in 17.06.x-ce ~ 17.06.2-ce.
The first several times "docker service logs" command successfully gives logs.
After scaling (decrease and increase) containers more than 10 times, the command doesn't work anymore.
$ docker info
Containers: 49
Running: 42
Paused: 0
Stopped: 7
Images: 153
Server Version: 17.06.2-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 769
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: ciagobmjzqaup5typm21rcnzu
Is Manager: true
ClusterID: 86ntnntsfirsvrhwhhzjuy8tv
Managers: 2
Nodes: 3
Orchestration:
Task History Retention Limit: 1
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 3
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Root Rotation In Progress: false
Node Address: xxxxx
Manager Addresses:
xxxxx:2377
xxxxx:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 3.13.0-79-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 15.67GiB
Name: xxxxx
ID: XNBD:OS6A:OXQ6:RF5F:KW5S:VIRI:QT7M:SKTC:IK3J:WGVI:4H3K:TP6M
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
0muzqhvzigqal85llqdhdxj64 xxxxx Ready Active Reachable
ciagobmjzqaup5typm21rcnzu * xxxxx Ready Active Leader
txoh28odun5kt42wuseo7qgdr xxxxx Ready Active
$ docker version
Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:00:33 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 19:59:26 2017
OS/Arch: linux/amd64
Experimental: false
53 remaining items
bluepuma77 commentedon Jul 25, 2022
Same here, service logs stopped working after heavy container rotation after network outage.
docker service logs -tf --since 24h <service>did not helpdocker swarm ca --rotatedid not helpdocker service update --force <service>did not helpAll bare metal servers running Debian, currently 115 Docker Swarm services, 475 containers
clay-moss commentedon Aug 12, 2022
@bluepuma77 Are you still experiencing this issue? Can you share a daemon stack trace from your managers if so please:
Itll be written to
/var/run/docker/Thanks
fiftin commentedon Oct 14, 2022
I have same issue with
docker service logs -f xxx.But
docker logs -f xxxfor same container works fine.RensTillmann commentedon Oct 26, 2022
I am no docker expert by any means, so don't ask me anything about it.
But it is a very anoying problem and I want to share what I do when it happens.
I sometimes use the below command, which does work, but it's anoying because on a cluster (3 nodes in my case) I have to log thre processes. Since the service is load balanced, but it does work.
For me to make the
docker service logswork on my swarm cluster (3 nodes) I have to demote theLeader.Then restart docker service on that node. Wait till it is ready and reachable.
Then promote to master again.
So the commands are along the lines of:
I run the below to make sure all the 3 nodes run the service
Perhaps it helps someone 👍
keltakmaster commentedon Jun 13, 2023
This did the job for me! Thanks
Radiergummi commentedon Nov 29, 2023
Has anyone since looked into this problem?
clay-moss commentedon Nov 29, 2023
@Radiergummi I believe this should be fixed by #42537
Available in Docker CE 20.10.8 and later: https://docs.docker.com/engine/release-notes/20.10/#20108
Qwarctick commentedon Dec 12, 2023
This does not fix the problem. I use v24.0.6 and still have the issue.
raysry commentedon Feb 20, 2024
24.0.7, still have the issue
pharapeti commentedon Mar 18, 2024
24.0.5 same issue
minakolta commentedon May 29, 2024
Any news about this issue ? It is happening as well in v24.0.6
mr-moon commentedon Oct 1, 2024
v27.2.0 - same issue. Loosing logs.
spiccrix commentedon Oct 12, 2024
Version 24.0.7: Same issue
The old task logs are only visible on the node where the shutdown task was running when executing docker service logs . If I demote the node and then re-promote it, I lose access to the logs, even when attempting to retrieve them from that same node.
drleon commentedon Jan 3, 2025
Same here:
mahnunchik commentedon May 25, 2025
Any news?