Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix json log format panic, change the flag name in flagIsSet #99257

Merged
merged 1 commit into from Mar 9, 2021

Conversation

chenyw1990
Copy link
Contributor

@chenyw1990 chenyw1990 commented Feb 20, 2021

What type of PR is this?

/kind bug

What this PR does / why we need it:

kubelet will panic with --logging-format=json

Which issue(s) this PR fixes:

Fixes #99239

Special notes for your reviewer:

@serathius
@ehashman
@knight42

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 20, 2021
@lala123912
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Feb 20, 2021
@chenyw1990
Copy link
Contributor Author

/retest

2 similar comments
@chenyw1990
Copy link
Contributor Author

/retest

@chenyw1990
Copy link
Contributor Author

/retest

@lala123912
Copy link
Contributor

/test pull-kubernetes-e2e-kind

@serathius
Copy link
Contributor

serathius commented Feb 20, 2021

/hold
This is not correct way to fix the issue. Json format doesn't support all klog flags, this logic is meant to protect against trying to enable JSON with one of the unsupported flags, so the logic is correct.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 20, 2021
@serathius
Copy link
Contributor

serathius commented Feb 20, 2021

#98524 looks like a culprit.
It changed names of flags that are returned by unsupportedLoggingFlags without updating flagIsSet function.

@serathius
Copy link
Contributor

Proposed fix, instead of adding json in Validate function, can you change flagIsSet function to

func flagIsSet(name string) bool {
	f := flag.Lookup(name)
	if f != nil {
		return f.DefValue != f.Value.String()
	}
	f = flag.Lookup(strings.Replace(name, "-", "_", -1))
	if f != nil {
		return f.DefValue != f.Value.String()
	}
	pf := pflag.Lookup(name)
	if pf != nil {
		return pf.DefValue != pf.Value.String()
	}
	panic(fmt.Sprintf("failed to lookup unsupported log flag: %s", name))
}

@chenyw1990 chenyw1990 changed the title add json to supported log format fix json log format panic, change the flag name in flagIsSet Feb 21, 2021
@chenyw1990
Copy link
Contributor Author

Proposed fix, instead of adding json in Validate function, can you change flagIsSet function to

func flagIsSet(name string) bool {
	f := flag.Lookup(name)
	if f != nil {
		return f.DefValue != f.Value.String()
	}
	f = flag.Lookup(strings.Replace(name, "-", "_", -1))
	if f != nil {
		return f.DefValue != f.Value.String()
	}
	pf := pflag.Lookup(name)
	if pf != nil {
		return pf.DefValue != pf.Value.String()
	}
	panic(fmt.Sprintf("failed to lookup unsupported log flag: %s", name))
}

sure, I have modify the code.

@chenyw1990
Copy link
Contributor Author

/retest

@serathius
Copy link
Contributor

/unhold
/lgtm
/approve

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 21, 2021
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 4, 2021
@ehashman
Copy link
Member

ehashman commented Mar 4, 2021

@chenyw1990 PTAL at the failing unit test, it looks related:

 === FAIL: cmd/kube-apiserver/app/options TestAddCustomGlobalFlags (0.00s)
    globalflags_test.go:59: [Default]: expected [add-dir-header alsologtostderr cloud-provider-gce-l7lb-src-cidrs cloud-provider-gce-lb-src-cidrs default-not-ready-toleration-seconds default-unreachable-toleration-seconds help log-backtrace-at log-dir log-file log-file-max-size log-flush-frequency logtostderr one-output skip-headers skip-log-headers stderrthreshold v vmodule], got [add_dir_header alsologtostderr cloud-provider-gce-l7lb-src-cidrs cloud-provider-gce-lb-src-cidrs default-not-ready-toleration-seconds default-unreachable-toleration-seconds help log-flush-frequency log_backtrace_at log_dir log_file log_file_max_size logtostderr one_output skip_headers skip_log_headers stderrthreshold v vmodule] 

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. area/apiserver and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 5, 2021
@ehashman
Copy link
Member

ehashman commented Mar 8, 2021

I think all comments should be addressed?

@liggitt would you PTAL?

@ehashman ehashman moved this from Waiting on Author to Needs Reviewer in Structured Logging Migration for Kubelet, 1.21 Mar 8, 2021
Copy link
Member

@liggitt liggitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a lot more coherent, thanks. Just had a couple suggestions about the unnormalize func

/approve
/hold for UnNormalize changes
instrumentation reviewer has lgtm once that is addressed

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 9, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chenyw1990, liggitt, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 9, 2021
@chenyw1990
Copy link
Contributor Author

@liggitt Thanks for your suggestions, I have modified the code based on the review comments.

@liggitt
Copy link
Member

liggitt commented Mar 9, 2021

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 9, 2021
@serathius
Copy link
Contributor

/hold cancel
as UnNormalize changes were addressed

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 9, 2021
@chenyw1990
Copy link
Contributor Author

/test pull-kubernetes-e2e-kind

@shubheksha shubheksha moved this from Needs Reviewer to Done in Structured Logging Migration for Kubelet, 1.21 Mar 9, 2021
@k8s-ci-robot k8s-ci-robot merged commit 4d969ac into kubernetes:master Mar 9, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Mar 9, 2021
@rifelpet
Copy link
Member

@chenyw1990 We're seeing failures in the kOps e2e jobs using CI builds of k8s due to the --log-file flag changing to --log_file

$ docker run -it --rm k8s.gcr.io/kube-apiserver:v1.20.4 -- kube-apiserver --help
...
      --log-file string                  If non-empty, use this log file
      --log-file-max-size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)

however recent CI builds show

$ curl https://storage.googleapis.com/kubernetes-release-dev/ci/v1.21.0-beta.1.200+2eb32e930a30ec/bin/linux/amd64/kube-apiserver.tar | docker load
$ docker run -it --rm k8s.gcr.io/kube-apiserver-amd64:v1.21.0-beta.1.200_2eb32e930a30ec -- kube-apiserver --help
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)

Is this intentional?

@chenyw1990
Copy link
Contributor Author

@rifelpet sorry, I have add a new PR to fixed this change. #100171

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

Kubelet panics when trying to set structured logging
8 participants