New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker with syslog driver option loses logs when the destination is down #30979

Open
sd65 opened this Issue Feb 13, 2017 · 4 comments

Comments

Projects
None yet
5 participants
@sd65

sd65 commented Feb 13, 2017

Description

Docker with syslog driver option does not retry to send logs when the destination is down.

Steps to reproduce the issue:

  1. Fire up a docker container using the syslog driver option
  2. Shutdown the remote syslog receiver
  3. Generate logs

Describe the results you received:

There is a line in journactl indicating the problem :

Feb 13 17:02:28 myHost docker[16878]: time="2017-02-13T17:02:28.967508497+01:00" level=error msg="my log" for logger syslog: dial tcp 192.xxx.xxx.xxx:5140: getsockopt: connection refused"

But no retries after, even when the destination becomes available.

Describe the results you expected:

I think, as syslog-ng, logs should be saved and docker should retry later.
Because, in my opinion, losing logs is not an option.

Additional information you deem important (e.g. issue happens only occasionally):

None

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:39:25 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:39:25 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
Images: 49
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-253:3-262148-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.507 GB
 Data Space Total: 107.4 GB
 Data Space Available: 17.99 GB
 Metadata Space Used: 3.973 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /xxx/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /xxx/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.3 MiB
Name: xxx
ID: H7FG:A4SC:EZFR:KT7H:FK3A:3EC2:KI7Y:G4P4:37IM:NTGI:D5X7:XMAD
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

RHEL 7, Physical

@imjacobclark

This comment has been minimized.

Show comment
Hide comment
@imjacobclark

imjacobclark Mar 7, 2017

@cpuguy83 Interested in picking this up... could anybody potentially mentor me through this?

imjacobclark commented Mar 7, 2017

@cpuguy83 Interested in picking this up... could anybody potentially mentor me through this?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Mar 7, 2017

Contributor

@imjacobclark Sure, the repo this would be under is github.com/RackSec/srslog/

Contributor

cpuguy83 commented Mar 7, 2017

@imjacobclark Sure, the repo this would be under is github.com/RackSec/srslog/

@tecnobrat

This comment has been minimized.

Show comment
Hide comment
@tecnobrat

tecnobrat Mar 15, 2017

There is lots more discussion here: #21966, this seems like a dupe of that?

tecnobrat commented Mar 15, 2017

There is lots more discussion here: #21966, this seems like a dupe of that?

@vroumvroum

This comment has been minimized.

Show comment
Hide comment
@vroumvroum

vroumvroum Aug 29, 2018

Hi. Same issue, it's really annoying.

Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:06 2017
OS/Arch: linux/amd64
Experimental: false

@tecnobrat : not quite the same issue. The one you mentionned is about launching a container when syslog endpoint is unavailable. In this case, the container starts properly but if by any chance the syslog endpoint becomes unavailable, you lose the messages (even in tcp).
I was expecting the message to be stored in the buffer while the syslog driver is unable to deliver it.

vroumvroum commented Aug 29, 2018

Hi. Same issue, it's really annoying.

Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:06:06 2017
OS/Arch: linux/amd64
Experimental: false

@tecnobrat : not quite the same issue. The one you mentionned is about launching a container when syslog endpoint is unavailable. In this case, the container starts properly but if by any chance the syslog endpoint becomes unavailable, you lose the messages (even in tcp).
I was expecting the message to be stored in the buffer while the syslog driver is unable to deliver it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment