Dockerd consumes and does not free memory after short period of unavailability of remote logging server

[glebsam]

Description
Dockerd consumes and does not free memory after short period of unavailability of remote logging server.

Steps to reproduce the issue:

1. Have some log storage server (reproduced with Graylog and Loki)
2. Run container specifying --log-driver= (reproduced with gelf and loki) and providing --log-opt mode=non-blocking --log-opt max-buffer-size=100m
3. Write down memory consumption of /usr/bin/dockerd
4. Restrict access to the remote log storage server (I used iptables or just stopped the server)
5. Watch growing memory consumption of /usr/bin/dockerd
6. Restore access to the remote log storage server
7. See that memory consumption stopped after a short period

Describe the results you received:
Memory consumption by /usr/bin/dockerd does not decreasing

Describe the results you expected:
Memory consumption by /usr/bin/dockerd decreasing

Additional information:

• Memory consumption speed depends of logs volume: more logs - quickier growth.
• Memory consumption by /usr/bin/dockerd does not decreased even after logging container was stopped and removed.
• Even after 12 hours nothing changed.
• Restart of docker engine frees memory, but the issue can be easily reproduced again.

Output of docker version:

Docker version 19.03.6-ce, build 369ce74

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 9
  Running: 1
  Paused: 0
  Stopped: 8
 Images: 14
 Server Version: 19.03.6-ce
 Storage Driver: devicemapper
  Pool Name: docker-docker--pool
  Pool Blocksize: 524.3kB
  Base Device Size: 10.74GB
  Backing Filesystem: ext4
  Udev Sync Supported: true
  Data Space Used: 2.517GB
  Data Space Total: 23.33GB
  Data Space Available: 20.81GB
  Metadata Space Used: 3.727MB
  Metadata Space Total: 25.17MB
  Metadata Space Available: 21.44MB
  Thin Pool Minimum Free Space: 2.333GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.135-RHEL7 (2016-11-16)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.173-106.229.amzn1.x86_64
 Operating System: Amazon Linux AMI 2018.03
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 959.2MiB
 Name: ip-10-0-4-93
 ID: NU7L:YQ3C:FCJV:CK5O:WBO7:OEJQ:52DY:34W4:HPQ5:GEL7:O6LK:CT3L
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS EC2 t3.micro, Operating system is Amazon Linux 1 (ECS-optimized)