CRI: Stop following container log when container exited. #44406

Random-Liu · 2017-04-12T20:30:01Z

This PR changed kubelet to periodically check whether container is running when following container logs, and stop following when container exited.

I've tried this PR in my local cluster:

Wed Apr 12 20:23:54 UTC 2017
Wed Apr 12 20:23:58 UTC 2017
Wed Apr 12 20:24:02 UTC 2017
Wed Apr 12 20:24:06 UTC 2017
Wed Apr 12 20:24:10 UTC 2017
Wed Apr 12 20:24:14 UTC 2017
Wed Apr 12 20:24:18 UTC 2017
Wed Apr 12 20:24:22 UTC 2017
Wed Apr 12 20:24:26 UTC 2017
Wed Apr 12 20:24:30 UTC 2017
Wed Apr 12 20:24:34 UTC 2017
Wed Apr 12 20:24:38 UTC 2017
Wed Apr 12 20:24:42 UTC 2017
Wed Apr 12 20:24:46 UTC 2017
failed to wait logs for log file "/var/log/pods/1d54634c7b31346fc3219f5e0b7507cc/nginx_0.log": container "b9a17a2c53550c3703ab350d85911743af8bf164a41813544fd08fb9585f7501" is not running (state="CONTAINER_EXITED")

The only difference is that ReadLogs will return error when container exits during following. I'm not sure whether we should get rid of it or not.

@yujuhong @feiskyer @JorritSalverda
/cc @kubernetes/sig-node-bugs

Release note:

CRI: `kubectl logs -f` now stops following when container stops, as it did pre-CRI.

k8s-reviewable · 2017-04-12T20:31:09Z

This change is

yujuhong · 2017-04-12T20:43:16Z

pkg/kubelet/kuberuntime/kuberuntime_logs.go

+			if err != nil {
+				return err
+			}
+			// Only keep following container log when it is running. There may be a race condition


If the container is not running, we can't tail it anyway. Do we need to care about that?

I thought we'd only reach this function if the container is running...

ACK. I tried with docker. docker logs -f will also return immediately if the container is in created state.

If so, I think we could also return here, we just shouldn't throw out the error.

yujuhong · 2017-04-12T20:50:20Z

pkg/kubelet/kuberuntime/kuberuntime_logs.go

+			// stateCheckPeriod here, it should be rare.
+			// Even when that happen, user could also retry after the container becomes running.
+			if s.State != runtimeapi.ContainerState_CONTAINER_RUNNING {
+				return fmt.Errorf("container %q is not running (state=%q)", id, s.State)


Don't think we need the error message.

Then I need to introduce another error to tell waitLogs not to continue parse log.

yujuhong · 2017-04-12T21:00:15Z

pkg/kubelet/kuberuntime/kuberuntime_logs.go

@@ -196,6 +204,40 @@ func ReadLogs(path string, apiOpts *v1.PodLogOptions, stdout, stderr io.Writer)
 	}
 }

+// waitLogs wait for the next log write. If the container is not running


Is this sentence complete? If the container is not running

Sorry, must be distracted by something else at that time...

mrunalp · 2017-04-13T02:21:16Z

Can't we continue to tail the logs even if container exited? The log file should be just there, right?

Random-Liu · 2017-04-13T05:54:26Z

@mrunalp We can still tail the log of exited container. However, we should not keep waiting for new lines after reading all the logs in following mode, because the container has exited already.

JorritSalverda · 2017-04-13T12:20:57Z

The behaviour prior to 1.6.0 was to stop following logs as soon as the container exited. With an exit code of 0, at least when a container terminated successfully. I used it to tail containers started by a job until successful completion.

Random-Liu · 2017-04-13T18:26:39Z

@yujuhong Updated the PR and get rid of the error.
Here is the new output:

$ cluster/kubectl.sh logs -f nginx-3
Thu Apr 13 18:23:42 UTC 2017
Thu Apr 13 18:23:46 UTC 2017
Thu Apr 13 18:23:50 UTC 2017
Thu Apr 13 18:23:54 UTC 2017
Thu Apr 13 18:23:58 UTC 2017
Thu Apr 13 18:24:02 UTC 2017
Thu Apr 13 18:24:06 UTC 2017
Thu Apr 13 18:24:10 UTC 2017
$

yujuhong · 2017-04-13T18:30:41Z

/lgtm

@Random-Liu could you add a release-note so that we can patch 1.6 later.

k8s-github-robot · 2017-04-13T18:52:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Random-Liu, yujuhong

Needs approval from an approver in each of these OWNERS Files:

~~pkg/kubelet/OWNERS~~ [Random-Liu,yujuhong]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

Random-Liu · 2017-04-14T00:09:54Z

@k8s-bot gce etcd3 e2e test this

k8s-github-robot · 2017-04-14T02:10:27Z

Automatic merge from submit-queue

@enisoc

…4406-upstream-release-1.6 Automatic merge from submit-queue Automated cherry pick of #44406 upstream release 1.6 Cherry pick of #44406 on release-1.6. #44406: CRI: Stop following container log when container exited. @enisoc **Release note**: ```release-note `kubectl logs -f` now stops following when container stops. ```

k8s-cherrypick-bot · 2017-04-14T18:51:31Z

Commit found in the "release-1.6" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

Random-Liu assigned yujuhong Apr 12, 2017

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 12, 2017

k8s-github-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-label-needed labels Apr 12, 2017

yujuhong reviewed Apr 12, 2017

View reviewed changes

Stop following container log when container exited.

2fbf34f

Random-Liu force-pushed the stop-following-when-exited branch from 80a8f75 to 2fbf34f Compare April 13, 2017 18:25

Random-Liu added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels Apr 13, 2017

yujuhong added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Apr 13, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 13, 2017

yujuhong added the cherrypick-candidate label Apr 13, 2017

yujuhong added this to the v1.6 milestone Apr 13, 2017

Random-Liu mentioned this pull request Apr 13, 2017

Automated cherry pick of #44406 upstream release 1.6 #44470

Merged

k8s-github-robot merged commit 1cf6ef0 into kubernetes:master Apr 14, 2017

Random-Liu deleted the stop-following-when-exited branch April 14, 2017 04:36

k8s-cherrypick-bot removed the cherrypick-candidate label Apr 14, 2017

bacongobbler mentioned this pull request Apr 19, 2017

kubernetes-1.6: build is never ends deis/builder#507

Closed

ncdc mentioned this pull request May 5, 2017

[builds][Slow] openshift pipeline build Pipeline with maven slave [It] should build and complete successfully openshift/origin#13984

Closed

gabemontero mentioned this pull request May 26, 2017

oc logs -f, oc start-builds --follow, hang openshift/origin#14148

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRI: Stop following container log when container exited. #44406

CRI: Stop following container log when container exited. #44406

Random-Liu commented Apr 12, 2017 •

edited by enisoc

Loading

k8s-reviewable commented Apr 12, 2017

yujuhong Apr 12, 2017

yujuhong Apr 12, 2017

Random-Liu Apr 12, 2017

Random-Liu Apr 13, 2017

yujuhong Apr 12, 2017

Random-Liu Apr 12, 2017

Random-Liu Apr 13, 2017

yujuhong Apr 12, 2017

Random-Liu Apr 12, 2017

Random-Liu Apr 13, 2017

mrunalp commented Apr 13, 2017

Random-Liu commented Apr 13, 2017 •

edited

Loading

JorritSalverda commented Apr 13, 2017

Random-Liu commented Apr 13, 2017

yujuhong commented Apr 13, 2017

k8s-github-robot commented Apr 13, 2017

Random-Liu commented Apr 14, 2017

k8s-github-robot commented Apr 14, 2017

k8s-cherrypick-bot commented Apr 14, 2017

CRI: Stop following container log when container exited. #44406

CRI: Stop following container log when container exited. #44406

Conversation

Random-Liu commented Apr 12, 2017 • edited by enisoc Loading

k8s-reviewable commented Apr 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrunalp commented Apr 13, 2017

Random-Liu commented Apr 13, 2017 • edited Loading

JorritSalverda commented Apr 13, 2017

Random-Liu commented Apr 13, 2017

yujuhong commented Apr 13, 2017

k8s-github-robot commented Apr 13, 2017

Random-Liu commented Apr 14, 2017

k8s-github-robot commented Apr 14, 2017

k8s-cherrypick-bot commented Apr 14, 2017

Random-Liu commented Apr 12, 2017 •

edited by enisoc

Loading

Random-Liu commented Apr 13, 2017 •

edited

Loading