New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix log tail with empty logs #36305

Merged
merged 1 commit into from Feb 14, 2018

Conversation

Projects
None yet
6 participants
@cpuguy83
Copy link
Contributor

cpuguy83 commented Feb 13, 2018

When tailing a container log, if the log file is empty it will cause the
log stream to abort with an unexpected EOF.
Note that this only applies to the "current" log file as rotated files
cannot be empty.

This fix just skips adding the "current" file the log tail if it is
empty.

Fixes #35370

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Feb 14, 2018

CI failing

20:33:03 --- FAIL: TestLogsFollowTailEmpty (0.44s)
20:33:03 	assertions.go:226: 
                          
	Error Trace:	logs_test.go:32
20:33:03 		
	Error:      	Received unexpected error:
20:33:03 		
	            	error from daemon in stream: Error grabbing logs: Error seeking for whence: 0, offset: 0
Fix log tail with empty logs
When tailing a container log, if the log file is empty it will cause the
log stream to abort with an unexpected `EOF`.
Note that this only applies to the "current" log file as rotated files
cannot be empty.

This fix just skips adding the "current" file the log tail if it is
empty.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>

@cpuguy83 cpuguy83 force-pushed the cpuguy83:35370_fix_logs_eof branch from 9854ace to f40860c Feb 14, 2018

@vdemeester
Copy link
Member

vdemeester left a comment

LGTM 🐸

@AkihiroSuda AkihiroSuda merged commit e698b6e into moby:master Feb 14, 2018

6 checks passed

dco-signed All commits are signed
experimental Jenkins build Docker-PRs-experimental 39290 has succeeded
Details
janky Jenkins build Docker-PRs 48035 has succeeded
Details
powerpc Jenkins build Docker-PRs-powerpc 8443 has succeeded
Details
windowsRS1 Jenkins build Docker-PRs-WoW-RS1 19565 has succeeded
Details
z Jenkins build Docker-PRs-s390x 8430 has succeeded
Details

@cpuguy83 cpuguy83 deleted the cpuguy83:35370_fix_logs_eof branch Feb 14, 2018

@bitterloa

This comment has been minimized.

Copy link

bitterloa commented Mar 19, 2018

Hi can someone clue me in as to how to fix this issue? I understand that this seems to be fixed in a particular branch but I'm not sure how to get this running locally. I've seen the Error grabbing logs: EOF across several versions of Docker for Mac--currenty I'm using the latest: Version 17.12.0-ce-mac55 (23011).

The issue occurs semi-randomly on multiple containers but for now in particular it happens for me on the official nginx and memcached containers. I can't test these containers locally because of lack of logs. Any pointers appreciated!

Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 471
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.60-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.818GiB
Name: linuxkit-025000000001
ID: P2DI:ESUG:JEHM:DIAM:Q2UH:VRKJ:VYLQ:C4NL:OW6X:Q4Q7:GUSO:NIDU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 102
 Goroutines: 136
 System Time: 2018-03-19T23:19:37.135769166Z
 EventsListeners: 3
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 20, 2018

@bitterloa this fix was merged in February, and thus is not yet in the Docker 17.12 release (which was released in December). The fix will be included in the upcoming 18.03 release of Docker (and is already in the latest release-candidate for 18.03; https://github.com/docker/docker-ce/blob/v18.03.0-ce-rc4/components/engine/daemon/logger/loggerutils/logfile.go#L195-L200)

@bitterloa

This comment has been minimized.

Copy link

bitterloa commented Mar 20, 2018

Thanks @thaJeztah -- what is the proper way for me to use the version that will fix this specific issue? Should I reinstall Docker to the edge version (which looks like it will uninstall stable version)? Is there any other way other than completely uninstalling Docker for me to get this to work? Thanks!

image

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 20, 2018

If you switch to the "edge" channel, you probably also get release-candidates (but be aware that those are not final releases, so may have bugs).

Otherwise, if you stick to the "stable" channel, you'll automatically get notified when the 18.03 release is out.

@bitterloa

This comment has been minimized.

Copy link

bitterloa commented Mar 20, 2018

I prefer to stick with the stable version--is there any way I can use the fix provided in 18.03 now though? If so, could you point me in the direction to use this particular fix until it is officially released? Is using the edge version the only way?

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 20, 2018

Yup, either stable, and wait until it’s released, or edge and get the fix earlier, but possibly have outstanding issues that still have to be addressed before the final release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment