Docker log rotation is not working the container log keeps on growing

[avi-kalar]

I have an issue that even though I set the docker log configuration it does not kick in and my docker logs grows till I run out of disk space. Can you please tell me when the log rotation is cleaned up, I have the following config set for my container when I run

docker inspect <container-id>

      `"LogConfig": {
            "Type": "json-file",
            "Config": {
                "max-file": "5",
                "max-size": "100m"
            }
        }`

and my log size is above 400 MB right now, but the log rotation hasn't kicked in.

Steps to reproduce the issue:

1. docker run --name badguy --log-opt max-size=1k --log-opt max-file=5 -d ubuntu sh -c "while true; do printf A; done"
2. wait for a while
3. docker kill badgu

Describe the results you received:
my log size is above 1k right now, but the log rotation hasn't kicked in.

Describe the results you expected:
Log rotation with file size not exceeding 1k

Output of docker version:

Output of docker info:

Containers: 39
 Running: 1
 Paused: 0
 Stopped: 38
Images: 1441
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1214
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.13-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952 GiB
Name: moby
ID: 6GE7:EIWS:LOD2:IFS2:3UHV:A62R:FWGU:RYUM:LTSJ:7D4B:VWES:DLEL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 23
 Goroutines: 32
 System Time: 2018-06-26T10:22:08.783714835Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[cpuguy83]

How are you checking the log size?

[kavinash-athena]

by running the command
sh -c "du -ch /var/lib/docker/containers/*/*-json.log"
and the output I get is

24K	/var/lib/docker/containers/40ab2ef8b7c01f5d41d672b064fc34fcaeb51c8d2c35285e7aaf3c3f48485baa/40ab2ef8b7c01f5d41d672b064fc34fcaeb51c8d2c35285e7aaf3c3f48485baa-json.log
390M	/var/lib/docker/containers/c074baa1bd4879bee699a636ddc681884ff85b6cf5b6203cc1d5a8d75bae0e94/c074baa1bd4879bee699a636ddc681884ff85b6cf5b6203cc1d5a8d75bae0e94-json.log
8.0K	/var/lib/docker/containers/dfea013ce11098384eb4861c53ad1df3f4cf72af07fccacd5e8531f4af34e0ee/dfea013ce11098384eb4861c53ad1df3f4cf72af07fccacd5e8531f4af34e0ee-json.log
390M	total

[thaJeztah]

17.03.1 is a really old version of Docker; are you still seeing the problem if you upgrade to a current version?

[kavinash-athena]

@thaJeztah - I have an EC2 instance with the following version

docker version
Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.9.4
 Git commit:	3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
 Built:	Tue Apr  3 23:37:44 2018
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.1-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.4
  Git commit:	7390fc6/17.12.1-ce
  Built:	Tue Apr  3 23:38:52 2018
  OS/Arch:	linux/amd64
  Experimental:	false

Will it work on this version, and do you know how often does the job for log rotation gets triggered?

[kavinash-athena]

@thaJeztah the test container is rotating logs as expected in the above version, but I still face issues with the container I have created for my application. I am not sure why I get different behaviour in this case even though the configs are similar and the only difference is that I set the log config via ECS task

            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "5",
                    "max-size": "100m"
                }
            },

Do you see anything wrong with the config ?

The config set on ECS task definition is

      "logConfiguration": {
        "logDriver": "json-file",
        "options": {
          "max-size": "100m",
          "max-file": "5"
        }

[cpuguy83]

Is this the config that's actually set on the container in question in Docker?

[kavinash-athena]

yes @cpuguy83 this is the config that I get when run docker inspect <container-id> for the application that I wanted to have log rotation enabled. The output for list command for the container logs are

sudo ls -la /var/lib/docker/containers/63d229839904d91dc22047dc22696c4297dd7bbf7fdb50b239d6ac6b5c72624b/
total 201800
drwx------ 4 root root      4096 Jun 27 08:54 .
drwx------ 6 root root      4096 Jun 27 08:59 ..
-rw-r----- 1 root root 206595504 Jun 28 05:31 63d229839904d91dc22047dc22696c4297dd7bbf7fdb50b239d6ac6b5c72624b-json.log
drwx------ 2 root root      4096 Jun 27 08:54 checkpoints
-rw------- 1 root root      5995 Jun 27 08:54 config.v2.json
-rw-r--r-- 1 root root      1292 Jun 27 08:54 hostconfig.json
-rw-r--r-- 1 root root        13 Jun 27 08:54 hostname
-rw-r--r-- 1 root root       174 Jun 27 08:54 hosts
-rw-r--r-- 1 root root       130 Jun 27 08:54 resolv.conf
-rw-r--r-- 1 root root        71 Jun 27 08:54 resolv.conf.hash
drwxrwxrwt 2 root root        40 Jun 27 08:54 shm

The max-size configured for this container is 100m but the file has exceeded that.

[kavinash-athena]

@cpuguy83 This is working now for a smaller max-size limit, the issue was the way I have been expanding the logs to reach 100m. I was appending the same file to the end of the file and thus growing the log files. Something wasn't quite adding up. Any guess what could have gone wrong here?

[thaJeztah]

You cannot change the configuration for existing containers; when you mention "expanding the logs"; were you trying to change the configuration after containers were already started with the old configuration? Are your tools attempting to change/update the configuration of existing containers?

[cpuguy83]

Ok, so maybe some misunderstanding of these params here.
max-size is max size per file, max-file is the maximum number of files.

[wouterds]

Can confirm that it does not work as expected.

[kavinash-athena]

No, this is not working as expected, and the config is right. If you had tried to simulate this, it wouldn’t have worked though. If it is just the containers pushing logs, then the log rotation is not working as expected.

…

-- Thanks Avinash From: Wouter De Schuyter <notifications@github.com> Reply-To: docker/cli <reply@reply.github.com> Date: Monday, 8 October 2018 at 4:02 AM To: docker/cli <cli@noreply.github.com> Cc: Avinash Kalarikkal <kavinash@athenahealth.com>, Comment <comment@noreply.github.com> Subject: Re: [docker/cli] Docker log rotation is not working the container log keeps on growing (#1148) Can confirm that it does not work as expected. [screen shot 2018-10-08 at 00 30 45]<https://user-images.githubusercontent.com/1210628/46587770-7d890300-ca91-11e8-842a-fe0660e273d8.jpg> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1148 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVK1o2dPIcKms8TL7heGUQNsOc0u29nUks5uioDcgaJpZM4U4VeR>.

[lucas-dehandschutter]

Hello,

I have a similar issue but by setting these value in the /etc/docker/daemon.json:

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "log-level": "info",
  "iptables": true,
  "insecure-registries": [],
  "cgroup-parent": "docker",
  "icc": false
}

Even after rebooting the server and redeploying all my docker services, the logs file of each services keeps growing with no limit.

My Docker version:

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:23 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[miande-acxiom]

Any update on this issue?

[widovsky]

It's not possible to change the log driver. To change it, you need to rerun the container.