Allows to combine the `-f` and `-l` flags in kubectl logs #67573

m1kola · 2018-08-19T21:11:07Z

What this PR does / why we need it:

This PR allows to combine the -f and -l flags in the kubectl logs by reading logs from multiple sources simultaneously. It is a part of what was requested in #52218 (without the part about -c).

Example:

kubectl logs -l app=logging_test --all-containers -f

Release note:

It is now possible to combine the `-f` and `-l` flags in `kubectl logs`

/sig cli
/kind feature

neolit123 · 2018-08-20T03:45:59Z

/ok-to-test

m1kola · 2018-08-21T13:41:06Z

/assign @liggitt

pkg/kubectl/cmd/logs.go

liggitt · 2018-08-21T14:01:43Z

pkg/kubectl/cmd/logs.go

-		}
+		go func(request *rest.Request) {
+			if err := o.ConsumeRequestFn(request, pipew); err != nil {
+				pipew.CloseWithError(err)


this closes all streams on first error. is that what we want?

Yes, I think this is the closest to what we currently have. At the moment, it's possible to get multiple log sources using a command like kubectl logs -l app=logging_test --all-containers, but this code will fail on the first error:

kubernetes/pkg/kubectl/cmd/logs.go

Lines 296 to 300 in 30e4f52

for _, request := range requests {

if err := o.ConsumeRequestFn(request, o.Out); err != nil {

return err

}

}

So, I think, the concurrent version should behave in a similar way: fail on the first error. At least for now. We can do something like retries as a separate improvements, I think. But I'm open for suggestions.

m1kola

@liggitt, thanks for the review. I've updated the PR with changes (see the fixup commits since your review) and replied to your comments.

Please take another look.

pkg/kubectl/cmd/logs.go

m1kola · 2018-08-21T17:10:21Z

pkg/kubectl/cmd/logs.go

-		}
+		go func(request *rest.Request) {
+			if err := o.ConsumeRequestFn(request, pipew); err != nil {
+				pipew.CloseWithError(err)


Yes, I think this is the closest to what we currently have. At the moment, it's possible to get multiple log sources using a command like kubectl logs -l app=logging_test --all-containers, but this code will fail on the first error:

kubernetes/pkg/kubectl/cmd/logs.go

Lines 296 to 300 in 30e4f52

for _, request := range requests {

if err := o.ConsumeRequestFn(request, o.Out); err != nil {

return err

}

}

So, I think, the concurrent version should behave in a similar way: fail on the first error. At least for now. We can do something like retries as a separate improvements, I think. But I'm open for suggestions.

pkg/kubectl/cmd/logs.go

m1kola · 2018-08-25T11:43:51Z

@liggitt tests are passing now. Could you, please, take another look after the weekend?

Changes since your previous review: ba7cb09bcc36c1eb6acf398e9c65d4afefd1df40...e5549129c0db54659711e71317b3b3ed480ae65d

In these commits:

A channel was replaced with a WaitGroup which helps to close PipeWriter once all log sources are exhausted
DefaultConsumeRequest was modified to make sure that it doesn't interleave sub-line when running concurrently
DefaultConsumeRequest was covered with unit tests

I'll squash commits before merging this

liggitt

one comment on error detection/handling on failed writes.

I'm still on the fence whether we want to support fan-out of long-running requests like this. @kubernetes/sig-cli-pr-reviews @kubernetes/sig-scalability-pr-reviews, any thoughts on that?

pkg/kubectl/cmd/logs.go

m1kola · 2018-09-03T20:18:52Z

/retest

m1kola · 2018-09-25T10:13:28Z

Hi @liggitt, is there anything I can do to move this forward? I'm open for further feedback and ready to discuss alternative approaches to implement this.

liggitt · 2018-09-26T05:37:21Z

I'd recommend raising the question with the CLI and scalability teams (either in slack, or on their mailing lists, or in one of their sig meetings) and see what feedback you get

pwittrock · 2018-09-28T16:44:57Z

@m1kola please put this on the agenda for the next sig-cli meeting (1 week from Wednesday.)

soltysh

Let's discuss in depth this on the next sig meeting on Oct 10th.

pkg/kubectl/cmd/logs.go

m1kola · 2018-10-11T17:31:34Z

Hi. Yesterday, after the sig-cli meeting I:

e77d1c1556aff5d27beae7b6f06cd8bbc156f58b - rebased the PR on top of the master branch
2255ba878e3c3f414ca0a323c2428d7a4e6de1c7 - added the flag that allows to control the number of concurrent of log streams with a (low) default limit

We also discussed the need to add the --prefix flag that will add source name to the log output. Everyone seemed to agree to handle this in a separate PR (I'll look into it once we are happy with this one).

pkg/kubectl/cmd/logs/logs.go

m1kola · 2018-11-17T22:36:12Z

/retest

m1kola · 2018-11-20T20:15:11Z

@soltysh and @seans3 I think, I've addressed the feedback you gave during the meeting. Could you, please, take another look at this PR and let me know, if you have any further comments.

I'm now looking into prefixing log lines with source (pod & container name) as we discussed, but it will be a separate PR to keep things simpler for reviewers and for me.

/assign @soltysh
/assign @seans3

soltysh

I left you some comments

soltysh · 2019-01-17T14:08:51Z

pkg/kubectl/cmd/logs/logs.go

@@ -151,6 +158,7 @@ func NewCmdLogs(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.C
 	cmd.Flags().StringVarP(&o.Container, "container", "c", o.Container, "Print the logs of this container")
 	cmdutil.AddPodRunningTimeoutFlag(cmd, defaultPodLogsTimeout)
 	cmd.Flags().StringVarP(&o.Selector, "selector", "l", o.Selector, "Selector (label query) to filter on.")
+	cmd.Flags().IntVar(&o.MaxFollowConcurency, "max-follow-concurency", o.MaxFollowConcurency, "Specify maximum number of concurrent logs to follow when using by a selector. Defaults to 5.")


I'd propose naming this max-requests

Maximum number of parallel log requests when using selector. Defaults to 5.

We could settle for a name in the middle, maybe something like max-log-requests

@soltysh @juanvallejo I'm not sure that about max-log-requests: sound too vague to me. When you do something like kubectl logs deployment/some-name, it will basically create N requests, but kubectl will be reading them sequentially. EDIT: Fixed call example. Also clarification: N is a number of sources from the deployment.

Probably it's ok for end-users, because they don't care about the number of sequentially requests, right?

soltysh · 2019-01-17T14:11:42Z

pkg/kubectl/cmd/logs/logs.go

+	if o.Follow && len(requests) > 1 {
+		if len(requests) > o.MaxFollowConcurency {
+			return fmt.Errorf(
+				"you are attempting to follow %d log streams, but maximum allowed concurency is %d. Use --max-follow-concurency to increase the limit",


Update when renaming the flag and use comma instead of a dot in that sentence.

soltysh · 2019-01-17T14:12:22Z

pkg/kubectl/cmd/logs/logs.go

+}
+
+func (o LogsOptions) concurrentConsumeRequest(requests []rest.ResponseWrapper) error {
+	piper, pipew := io.Pipe()


reader and writer are better variable names

soltysh · 2019-01-17T14:13:04Z

pkg/kubectl/cmd/logs/logs.go

+	return o.sequentialConsumeRequest(requests)
+}
+
+func (o LogsOptions) concurrentConsumeRequest(requests []rest.ResponseWrapper) error {


parallelConsumeRequest

soltysh · 2019-01-17T14:24:44Z

pkg/kubectl/cmd/logs/logs.go

+		}(request)
+	}
+
+	go func() {


I think you want to block the main flow until you hear back from the wait group, iow. this code should not be part of any goroutine.

io.Copy below blocks the main flow. If we do not close pipe writer (pipew currently), it will be blocking the main flow forever (even if the server closed the connection). So we are waiting for all requests to finish in a separate goroutine and close the writer (as a result io.Copy stops blocking the main flow).

Or do I miss something?

soltysh · 2019-01-17T14:27:55Z

pkg/kubectl/cmd/logs/logs.go

+			return err
+		}
+
+		if err != nil {


if err != nil && err != io.EOF { return err } return nil

@soltysh return nil will stop the for loop after the first iteration. We want to return from the function when the first error appears, but we don't treat io.EOF as an error (because it is what we are waiting for). In case of no error we want to continue the loop.

juanvallejo · 2019-01-17T15:35:55Z

@soltysh @m1kola how about naming the flag something like max-log-requests? At least to be a bit more descriptive about its intention

m1kola · 2019-01-20T13:18:39Z

/test pull-kubernetes-integration

Which makes possible to combile the `-f` and `-l` flags in kubectl logs

m1kola · 2019-01-20T15:22:25Z

/retest

m1kola · 2019-01-20T15:58:05Z

@soltysh please, take another look. I addressed your feedback in the latest changes and in comments.

And thanks for the review!

soltysh

/lgtm
/approve
@m1kola thanks for your patience 👍

k8s-ci-robot · 2019-02-25T19:12:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: m1kola, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubectl/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

m1kola · 2019-02-25T20:20:45Z

Test failures seem to be unrelated so let's try to rerun them.
/retest

k8s-ci-robot requested review from dshulyak and smarterclayton August 19, 2018 21:11

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 19, 2018

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 20, 2018

k8s-ci-robot assigned liggitt Aug 21, 2018

liggitt reviewed Aug 21, 2018

View reviewed changes