Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV in github.com/fluent/fluent-logger-golang/fluent.(*Fluent).write #40829

Closed
zhy1 opened this issue Apr 16, 2020 · 3 comments · Fixed by #42418
Closed

SIGSEGV in github.com/fluent/fluent-logger-golang/fluent.(*Fluent).write #40829

zhy1 opened this issue Apr 16, 2020 · 3 comments · Fixed by #42418
Labels
area/logging exp/beginner kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.

Comments

@zhy1
Copy link

zhy1 commented Apr 16, 2020

Description
docker crash

Steps to reproduce the issue:

  1. i use fluentd

Describe the results you received:
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x563fc57ebad9]
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: goroutine 561046 [running]:
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).write(0xc000824210, 0xc00153f350, 0xc00153f2f0, 0xc00153f320)
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:384 +0x119
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).postRawData(0xc000824210, 0xc00153f350, 0x44, 0x18f43bbd)
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:223 +0x6e
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).EncodeAndPostData(0xc000824210, 0xc0005e4b90, 0x44, 0x18f43bbd, 0xed62a3596, 0x0, 0x563fc6792da0, 0xc00153f2f0, 0x98, 0x8)
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:210 +0x1be
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent.(*Fluent).PostWithTime(0xc000824210, 0xc0005e4b90, 0x44, 0x18f43bbd, 0xed62a3596, 0x0, 0x563fc67982c0, 0xc00153f2c0, 0x1a13, 0x563fc41b1833)
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fluent/fluent-logger-golang/fluent/fluent.go:201 +0x7bb
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: github.com/docker/docker/daemon/logger/fluentd.(*fluentd).Log(0xc001717ec0, 0xc0012ac230, 0x563fc7fa82a0, 0x400a)
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/logger/fluentd/fluentd.go:177 +0x3cb
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: github.com/docker/docker/daemon/logger.(*Copier).copySrc(0xc001576040, 0x563fc5ac17b5, 0x6, 0x7fd920d88d70, 0xc001717f00)
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/logger/copier.go:123 +0x369
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: created by github.com/docker/docker/daemon/logger.(*Copier).Run
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ dockerd[2120]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/logger/copier.go:48 +0x111
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ systemd[1]: Unit docker.service entered failed state.
Apr 16 19:16:38 iZuf6jb9uv1lmingd0agruZ systemd[1]: docker.service failed.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: docker.service holdoff time over, scheduling restart.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: Stopped Docker Application Container Engine.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: Closed Docker Socket for the API.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: Stopping Docker Socket for the API.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: Starting Docker Socket for the API.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: Listening on Docker Socket for the API.
Apr 16 19:16:40 iZuf6jb9uv1lmingd0agruZ systemd[1]: Starting Docker Application Container Engine...

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

 docker info
Client:
 Debug Mode: false

Server:
 Containers: 17
  Running: 15
  Paused: 0
  Stopped: 2
 Images: 59
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-957.21.3.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.638GiB
 Name: iZuf6jb9uv1lmingd0agruZ
 ID: TUDV:GL7K:XGNU:AN7J:Z3X7:KRGN:D2K3:JP5Y:FGVZ:ZOWA:GQR3:7L3G
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
ALIYUN
@AkihiroSuda AkihiroSuda added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. area/logging labels Apr 19, 2020
@AkihiroSuda AkihiroSuda changed the title docker crash SIGSEGV in github.com/fluent/fluent-logger-golang/fluent.(*Fluent).write Apr 19, 2020
@libnine
Copy link

libnine commented May 21, 2020

Is this a fluentd error in write ? Happy to take a look at it either way.

@akerouanton
Copy link
Member

akerouanton commented May 21, 2020 •
edited

Docker v19.03.1 uses fluent/fluent-logger-golang v1.4.0 (see here). fluent/fluent-logger-golang v1.5.0 has been released on Feb. 19th and by looking at the diff, I found out this commit which fixes the exact problem described here.

Unfortunately Docker entered its code freeze period to prepare the next major release, so there're little chances this got fixed in that release. However, the fluent-logger-golang lib, and thus the fluentd driver in Docker, are suffering from another (major) bug, as reported in #40063.

Docker maintainers currently recommend to create and use a log driver plugin to bump fluent-logger lib to a newer, fixed version. The good news is, I worked on a fix for #40063 and created my own logging driver plugin to overcome these issues and I just published a v0.1.

@cpuguy83
Copy link
Member

Feel free to make a pr to update fluentd.

@sparrc sparrc mentioned this issue May 25, 2021
Merged
sparrc added a commit to sparrc/moby that referenced this issue May 25, 2021
@sparrc
vendor: github.com/fluent/fluent-logger-golang 1.6.1
a6a98d6 
Updates the fluent logger library. Namely this fixes a couple places
where the library could panic when closing and writing to channels.

see fluent/fluent-logger-golang#93 and
fluent/fluent-logger-golang#95

closes moby#40829
closes moby#32567

Signed-off-by: Cam <gh@sparr.email>
PettitWesley pushed a commit to PettitWesley/moby that referenced this issue Jan 13, 2022
@sparrc @PettitWesley
vendor: github.com/fluent/fluent-logger-golang 1.6.1
d6f3add 
Updates the fluent logger library. Namely this fixes a couple places
where the library could panic when closing and writing to channels.

see fluent/fluent-logger-golang#93 and
fluent/fluent-logger-golang#95

closes moby#40829
closes moby#32567

Signed-off-by: Cam <gh@sparr.email>
(cherry picked from commit a6a98d6)
Signed-off-by: Wesley <wppttt@amazon.com>
@thaJeztah thaJeztah mentioned this issue Jan 13, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/logging exp/beginner kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vendor: github.com/fluent/fluent-logger-golang 1.6.1 sparrc/moby
5 participants
@akerouanton @cpuguy83 @AkihiroSuda @zhy1 @libnine