add --logging-format flag to kubelet #91532

AfrouzMashayekhi · 2020-05-28T13:51:02Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
Ref kubernetes/enhancements#1602

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

kubelet: add '--logging-format' flag to support structured logging

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging

k8s-ci-robot · 2020-05-28T13:51:10Z

Welcome @afrouzMashaykhi!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2020-05-28T13:51:11Z

Hi @afrouzMashaykhi. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

AfrouzMashayekhi · 2020-05-28T13:54:39Z

/cc @serathius

serathius · 2020-05-28T14:03:25Z

Hey @afrouzMashaykhi Thanks for contributing!
Adding flag to Kubelet is more complicated due support to dynamic configuration. We need to add LoggingFormat field to KubeletConfig object and update all API generation.
Exmple. #85282

mtaufen · 2020-05-28T23:29:38Z

cmd/kubelet/app/options/options.go

 	}
-
+	errs = append(errs, f.Logs.Validate()...)


If the validation passes here, would this append a nil error to the slice, and result in a validation failure further up?

I changed validation of logformat

mtaufen · 2020-05-28T23:41:36Z

cmd/kubelet/app/server.go

@@ -500,6 +500,8 @@ func run(s *options.KubeletServer, kubeDeps *kubelet.Dependencies, featureGate f
 	if err != nil {
 		klog.Errorf("unable to register KubeletConfiguration with configz, error: %v", err)
 	}
+	// set klog of kubelet
+	s.Logs.Apply()


Applying the config to global settings should probably happen super-early in the bootstrap.
We can start writing logs almost immediately:

kubernetes/cmd/kubelet/app/server.go

Lines 152 to 164 in 0891f69

Run: func(cmd *cobra.Command, args []string) {

// initial flag parse, since we disable cobra's flag parsing

if err := cleanFlagSet.Parse(args); err != nil {

cmd.Usage()

klog.Fatal(err)

}

// check if there are non-flag arguments in the command line

cmds := cleanFlagSet.Args()

if len(cmds) > 0 {

cmd.Usage()

klog.Fatalf("unknown command: %s", cmds[0])

}

Also, are any additional changes required wrt the InitLogs call in https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/kubelet.go#L38?

Based on KEP the logging format flag just sets the format of logging and I think, changing log messages handled by new klog methods. @serathius please correct me if I am wrong.

Michael are you proposing then that we move this --logging-format flag init/setup/apply to InitLogs? This would mean that we would need to make this flag global.
In #89683 we have implemented it as standard flag initialized with component. Like you mentioned that it has problem that we invoke klog before we initialize components.

Pros of global flag:

Consistent logging configuration (as initialization happens before component bootstrap)

Cons:

All logging flags will need to be kept as global flag

LoggerConfig cannot be added to Components (no dynamic config, no control for component owner about this flag)

I think that having consistent log format is important so we should go with global flag. This would mean that all PRs adding LoggingConfig are no longer needed :(. WDYT?

Not necessarily into InitLogs - just that logging config should probably be set before any log messages are written?

mtaufen · 2020-05-28T23:46:17Z

And as @serathius mentioned, there are a few additional changes required to support the kubelet config file. You can find details in our contributor guide for migrating flags to config: https://docs.google.com/document/d/10Kz__miCWEOekC6WnsUdxYWSLrf-t1AkHHpZN57ei5c/edit

serathius · 2020-05-29T11:03:05Z

/ok-to-test

serathius · 2020-07-01T14:37:16Z

/milestone v1.19

liggitt · 2020-07-01T15:18:02Z

to fix the fuzzer test, update pkg/kubelet/apis/config/fuzzer/fuzzer.go to match the default logging format:

--- a/pkg/kubelet/apis/config/fuzzer/fuzzer.go
+++ b/pkg/kubelet/apis/config/fuzzer/fuzzer.go
@@ -99,6 +99,9 @@ func Funcs(codecs runtimeserializer.CodecFactory) []interface{} {
 			obj.ConfigMapAndSecretChangeDetectionStrategy = "Watch"
 			obj.AllowedUnsafeSysctls = []string{}
 			obj.VolumePluginDir = kubeletconfigv1beta1.DefaultVolumePluginDir
+			if obj.Logging.Format == "" {
+				obj.Logging.Format = "text"
+			}
 		},
 	}
 }

liggitt · 2020-07-01T15:18:18Z

/approve

lgtm once the fuzzer test is fixed

k8s-ci-robot · 2020-07-01T15:19:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afrouzMashaykhi, liggitt, mtaufen, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [liggitt,wojtek-t]
~~cmd/kubelet/OWNERS~~ [liggitt,wojtek-t]
~~pkg/kubelet/apis/config/OWNERS~~ [liggitt,mtaufen,wojtek-t]
~~staging/src/k8s.io/component-base/config/OWNERS~~ [liggitt]
~~staging/src/k8s.io/kubelet/OWNERS~~ [liggitt,mtaufen,wojtek-t]
~~staging/src/k8s.io/kubelet/config/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

serathius · 2020-07-01T18:57:02Z

/retest

serathius · 2020-07-01T19:16:28Z

/retest

serathius · 2020-07-01T19:35:54Z

/lgtm

AfrouzMashayekhi · 2020-07-01T20:34:43Z

/retest

fejta-bot · 2020-07-02T06:16:35Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

serathius · 2020-07-02T07:02:39Z

/retest

AfrouzMashayekhi · 2020-07-02T08:51:32Z

/retest

serathius · 2020-07-02T09:27:40Z

/priority important-soon

AfrouzMashayekhi · 2020-07-02T10:54:09Z

/retest

k8s-ci-robot · 2020-07-02T13:35:59Z

@afrouzMashaykhi: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kind	`b92b04e`	link	`/test pull-kubernetes-e2e-kind`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

serathius · 2020-07-02T13:37:01Z

/retest

k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 28, 2020

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 28, 2020

k8s-ci-robot requested review from mtaufen and yifan-gu May 28, 2020 13:52

k8s-ci-robot requested a review from serathius May 28, 2020 13:54

serathius mentioned this pull request May 28, 2020

Structured logging kubernetes/enhancements#1602

Open

39 tasks

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 28, 2020

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 28, 2020

AfrouzMashayekhi force-pushed the log-flag-kubelet branch from cdb5d5a to 3d32a4e Compare May 28, 2020 19:52

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. labels May 28, 2020

mtaufen reviewed May 28, 2020

View reviewed changes

k8s-ci-robot added this to the v1.19 milestone Jul 1, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 1, 2020

AfrouzMashayekhi added 3 commits July 1, 2020 20:07

add --logging-format flag to kubelet

ba8189a

add loggingConfig struct to kubelet config

ac8d2e8

add LoggingConfiguration struct to component-base/config

3ab25f1

AfrouzMashayekhi force-pushed the log-flag-kubelet branch from 18a5ad4 to 734c240 Compare July 1, 2020 15:47

add default for loggingconfiguration struct

b92b04e

AfrouzMashayekhi force-pushed the log-flag-kubelet branch from 734c240 to b92b04e Compare July 1, 2020 17:59

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 1, 2020

k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 2, 2020

k8s-ci-robot merged commit 13a4a71 into kubernetes:master Jul 2, 2020

github-actions bot mentioned this pull request Jul 8, 2020

Week Ending July 5, 2020 dev-obs/actus#190

Open

ari-becker mentioned this pull request May 23, 2021

Support structured logging ('--logging-format=json') kubernetes/kops#11576

Closed

cyclinder mentioned this pull request Feb 10, 2023

Introduce JSON logging format for kube-proxy #105165

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add --logging-format flag to kubelet #91532

add --logging-format flag to kubelet #91532

AfrouzMashayekhi commented May 28, 2020 •

edited

Loading

k8s-ci-robot commented May 28, 2020

k8s-ci-robot commented May 28, 2020

AfrouzMashayekhi commented May 28, 2020

serathius commented May 28, 2020

mtaufen May 28, 2020

AfrouzMashayekhi May 31, 2020

mtaufen May 28, 2020

AfrouzMashayekhi May 31, 2020 •

edited

Loading

serathius Jun 1, 2020

mtaufen Jun 1, 2020

mtaufen commented May 28, 2020

serathius commented May 29, 2020

serathius commented Jul 1, 2020

liggitt commented Jul 1, 2020

liggitt commented Jul 1, 2020

k8s-ci-robot commented Jul 1, 2020

serathius commented Jul 1, 2020

serathius commented Jul 1, 2020

serathius commented Jul 1, 2020

AfrouzMashayekhi commented Jul 1, 2020

fejta-bot commented Jul 2, 2020

serathius commented Jul 2, 2020

AfrouzMashayekhi commented Jul 2, 2020

serathius commented Jul 2, 2020

AfrouzMashayekhi commented Jul 2, 2020

k8s-ci-robot commented Jul 2, 2020

serathius commented Jul 2, 2020

	Run: func(cmd *cobra.Command, args []string) {
	// initial flag parse, since we disable cobra's flag parsing
	if err := cleanFlagSet.Parse(args); err != nil {
	cmd.Usage()
	klog.Fatal(err)
	}

	// check if there are non-flag arguments in the command line
	cmds := cleanFlagSet.Args()
	if len(cmds) > 0 {
	cmd.Usage()
	klog.Fatalf("unknown command: %s", cmds[0])
	}

add --logging-format flag to kubelet #91532

add --logging-format flag to kubelet #91532

Conversation

AfrouzMashayekhi commented May 28, 2020 • edited Loading

k8s-ci-robot commented May 28, 2020

k8s-ci-robot commented May 28, 2020

AfrouzMashayekhi commented May 28, 2020

serathius commented May 28, 2020

mtaufen May 28, 2020

Choose a reason for hiding this comment

AfrouzMashayekhi May 31, 2020

Choose a reason for hiding this comment

mtaufen May 28, 2020

Choose a reason for hiding this comment

AfrouzMashayekhi May 31, 2020 • edited Loading

Choose a reason for hiding this comment

serathius Jun 1, 2020

Choose a reason for hiding this comment

mtaufen Jun 1, 2020

Choose a reason for hiding this comment

mtaufen commented May 28, 2020

serathius commented May 29, 2020

serathius commented Jul 1, 2020

liggitt commented Jul 1, 2020

liggitt commented Jul 1, 2020

k8s-ci-robot commented Jul 1, 2020

serathius commented Jul 1, 2020

serathius commented Jul 1, 2020

serathius commented Jul 1, 2020

AfrouzMashayekhi commented Jul 1, 2020

fejta-bot commented Jul 2, 2020

serathius commented Jul 2, 2020

AfrouzMashayekhi commented Jul 2, 2020

serathius commented Jul 2, 2020

AfrouzMashayekhi commented Jul 2, 2020

k8s-ci-robot commented Jul 2, 2020

serathius commented Jul 2, 2020

AfrouzMashayekhi commented May 28, 2020 •

edited

Loading

AfrouzMashayekhi May 31, 2020 •

edited

Loading