Skip to content

docker service logs stops showing logs from containers on different nodes #33183

Open

Description

@benturner

Description
Running docker service logs foo on a swarm master where foo is a service with multiple replicas across different nodes eventually stops merging the logs from those other nodes. It seems to always work just fine right after the service is created.

Steps to reproduce the issue:

  1. Create a service foo with replicas across multiple nodes
  2. Run docker service logs --follow foo
  3. Initially observe logs from multiple containers across different nodes
  4. Go away and do something else for a while
  5. Run docker service logs --follow foo
  6. Observe old logs from multiple containers across different nodes but new logs only contain logs from the node on which you're running the command

Describe the results you received:
Logs from containers on the current node only

Describe the results you expected:
Logs from all containers on all nodes

Additional information you deem important (e.g. issue happens only occasionally):
Seems to work fine at first but then within some amount of time it stops working. I've tried with both json-file and journald log drivers.

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 7
 Running: 7
 Paused: 0
 Stopped: 0
Images: 6
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 57
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: active
 NodeID: jbsbgj3on5coa7f996rle8bpk
 Is Manager: true
 ClusterID: 7uzbzxfjt8nf6p18wbzv8ek84
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.16.0.5
 Manager Addresses:
  172.16.0.5:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.636GiB
Name: swarmm-master-94917428-0
ID: NNL7:YHDL:5ALU:4ZXF:J3BL:VAIV:UI2T:TV5U:UGQL:UCQC:WWCP:TQDO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 149
 Goroutines: 310
 System Time: 2017-05-12T22:02:26.629917059Z
 EventsListeners: 7
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on azure using an acs-engine template (https://github.com/Azure/acs-engine). Currently just testing this so I'm using one manager and one worker node. The replicas for my service get split over both nodes.

Activity

  1. thaJeztah commented on May 15, 2017

    @thaJeztah
    Member

    ping @dperny PTAL

  2. dperny commented on May 15, 2017

    @dperny
    Contributor

    Definitely a bug. @benturner about how long does it take for the logs to fall off? Also, since it's a test cluster, would it be possible for you to post daemon logs from about the time when you start the logs to the time when they fall off? If not, NBD, I will try to reproduce locally.

  3. benturner commented on May 15, 2017

    @benturner
    Author

    @dperny Yes I will try to grab some logs today. Thanks for looking into this.

  4. added
    kind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
    on May 15, 2017
  5. benturner commented on May 18, 2017

    @benturner
    Author

    Apologies, but I have tried and failed for a few days to get this to reproduce. I'll reopen if I can ever catch it. Thanks.

  6. thaJeztah commented on May 18, 2017

    @thaJeztah
    Member

    Thanks! Keep us posted 👍

  7. yunghoy commented on Sep 19, 2017

    @yunghoy

    It should be reopend. This issue reproduced in 17.06.x-ce ~ 17.06.2-ce.

    The first several times "docker service logs" command successfully gives logs.
    After scaling (decrease and increase) containers more than 10 times, the command doesn't work anymore.

    $ docker info

    Containers: 49
    Running: 42
    Paused: 0
    Stopped: 7
    Images: 153
    Server Version: 17.06.2-ce
    Storage Driver: aufs
    Root Dir: /var/lib/docker/aufs
    Backing Filesystem: extfs
    Dirs: 769
    Dirperm1 Supported: false
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
    Volume: local
    Network: bridge host macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
    Swarm: active
    NodeID: ciagobmjzqaup5typm21rcnzu
    Is Manager: true
    ClusterID: 86ntnntsfirsvrhwhhzjuy8tv
    Managers: 2
    Nodes: 3
    Orchestration:
    Task History Retention Limit: 1
    Raft:
    Snapshot Interval: 10000
    Number of Old Snapshots to Retain: 3
    Heartbeat Tick: 1
    Election Tick: 3
    Dispatcher:
    Heartbeat Period: 5 seconds
    CA Configuration:
    Expiry Duration: 3 months
    Force Rotate: 0
    Root Rotation In Progress: false
    Node Address: xxxxx
    Manager Addresses:
    xxxxx:2377
    xxxxx:2377
    Runtimes: runc
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
    runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
    init version: 949e6fa
    Security Options:
    apparmor
    Kernel Version: 3.13.0-79-generic
    Operating System: Ubuntu 14.04.5 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 6
    Total Memory: 15.67GiB
    Name: xxxxx
    ID: XNBD:OS6A:OXQ6:RF5F:KW5S:VIRI:QT7M:SKTC:IK3J:WGVI:4H3K:TP6M
    Docker Root Dir: /var/lib/docker
    Debug Mode (client): false
    Debug Mode (server): false
    Registry: https://index.docker.io/v1/
    Experimental: false
    Insecure Registries:
    127.0.0.0/8
    Live Restore Enabled: false

    WARNING: No swap limit support

    $ docker node ls

    ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
    0muzqhvzigqal85llqdhdxj64 xxxxx Ready Active Reachable
    ciagobmjzqaup5typm21rcnzu * xxxxx Ready Active Leader
    txoh28odun5kt42wuseo7qgdr xxxxx Ready Active

    $ docker version

    Client:
    Version: 17.06.2-ce
    API version: 1.30
    Go version: go1.8.3
    Git commit: cec0b72
    Built: Tue Sep 5 20:00:33 2017
    OS/Arch: linux/amd64

    Server:
    Version: 17.06.2-ce
    API version: 1.30 (minimum version 1.12)
    Go version: go1.8.3
    Git commit: cec0b72
    Built: Tue Sep 5 19:59:26 2017
    OS/Arch: linux/amd64
    Experimental: false

  8. 53 remaining items

  9. bluepuma77 commented on Jul 25, 2022

    @bluepuma77

    Same here, service logs stopped working after heavy container rotation after network outage.

    docker service logs -tf --since 24h <service> did not help

    docker swarm ca --rotate did not help

    docker service update --force <service> did not help

    All bare metal servers running Debian, currently 115 Docker Swarm services, 475 containers

    ID    HOSTNAME       STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
    1     server11       Ready     Active                          20.10.17
    2     server12       Ready     Active                          20.10.17
    3     server13       Ready     Active                          20.10.17
    4     server50       Ready     Active                          20.10.8
    5 *   server51       Ready     Active         Reachable        20.10.7
    6     server52       Ready     Active         Reachable        20.10.7
    7     server53       Ready     Active         Leader           20.10.7
    8     server54       Ready     Active                          20.10.6
    9     server55       Ready     Active                          20.10.6
    
  10. clay-moss commented on Aug 12, 2022

    @clay-moss

    @bluepuma77 Are you still experiencing this issue? Can you share a daemon stack trace from your managers if so please:

    Itll be written to /var/run/docker/

    Thanks

  11. fiftin commented on Oct 14, 2022

    @fiftin

    I have same issue with docker service logs -f xxx.

    But docker logs -f xxx for same container works fine.

  12. RensTillmann commented on Oct 26, 2022

    @RensTillmann

    I am no docker expert by any means, so don't ask me anything about it.
    But it is a very anoying problem and I want to share what I do when it happens.

    I sometimes use the below command, which does work, but it's anoying because on a cluster (3 nodes in my case) I have to log thre processes. Since the service is load balanced, but it does work.

    watch -t docker logs --tail 500 process_name.1.xxxxxxxxxxxxx
    

    For me to make the docker service logs work on my swarm cluster (3 nodes) I have to demote the Leader.
    Then restart docker service on that node. Wait till it is ready and reachable.
    Then promote to master again.

    So the commands are along the lines of:

    docker node demote managerX
    systemctl restart docker
    docker node promote managerX
    

    I run the below to make sure all the 3 nodes run the service

    docker service scale my_service=4
    docker service scale my_service=3
    

    Perhaps it helps someone 👍

  13. keltakmaster commented on Jun 13, 2023

    @keltakmaster

    I am no docker expert by any means, so don't ask me anything about it. But it is a very anoying problem and I want to share what I do when it happens.

    I sometimes use the below command, which does work, but it's anoying because on a cluster (3 nodes in my case) I have to log thre processes. Since the service is load balanced, but it does work.

    watch -t docker logs --tail 500 process_name.1.xxxxxxxxxxxxx
    

    For me to make the docker service logs work on my swarm cluster (3 nodes) I have to demote the Leader. Then restart docker service on that node. Wait till it is ready and reachable. Then promote to master again.

    So the commands are along the lines of:

    docker node demote managerX
    systemctl restart docker
    docker node promote managerX
    

    I run the below to make sure all the 3 nodes run the service

    docker service scale my_service=4
    docker service scale my_service=3
    

    Perhaps it helps someone 👍

    This did the job for me! Thanks

  14. Radiergummi commented on Nov 29, 2023

    @Radiergummi

    Has anyone since looked into this problem?

  15. clay-moss commented on Nov 29, 2023

    @clay-moss

    @Radiergummi I believe this should be fixed by #42537

    Available in Docker CE 20.10.8 and later: https://docs.docker.com/engine/release-notes/20.10/#20108

  16. Qwarctick commented on Dec 12, 2023

    @Qwarctick

    This does not fix the problem. I use v24.0.6 and still have the issue.

  17. raysry commented on Feb 20, 2024

    @raysry

    24.0.7, still have the issue

  18. pharapeti commented on Mar 18, 2024

    @pharapeti

    24.0.5 same issue

  19. minakolta commented on May 29, 2024

    @minakolta

    Any news about this issue ? It is happening as well in v24.0.6

  20. mr-moon commented on Oct 1, 2024

    @mr-moon

    v27.2.0 - same issue. Loosing logs.

  21. spiccrix commented on Oct 12, 2024

    @spiccrix

    Version 24.0.7: Same issue

    The old task logs are only visible on the node where the shutdown task was running when executing docker service logs . If I demote the node and then re-promote it, I lose access to the logs, even when attempting to retrieve them from that same node.

  22. drleon commented on Jan 3, 2025

    @drleon

    Same here:

    ID                            HOSTNAME                STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
    8guhyzo2unemlsso7tr2v1d3x *   server-01.localdomain   Ready     Active         Leader           27.3.1
    50qd9cgitn5tfxx3wcdksalmi     server-02.localdomain   Ready     Active                          27.3.1
    
    
  23. mahnunchik commented on May 25, 2025

    @mahnunchik

    Any news?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/swarmkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/17.05

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions