Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve logging on kube-proxy exit #120375

Merged
merged 1 commit into from Sep 10, 2023
Merged

Improve logging on kube-proxy exit #120375

merged 1 commit into from Sep 10, 2023

Conversation

pegasas
Copy link
Contributor

@pegasas pegasas commented Sep 2, 2023 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #120355

Special notes for your reviewer:

kube-proxy don't panic on exit when the Node object changes its PodCIDR

Does this PR introduce a user-facing change?

No

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

https://github.com/kubernetes/kubernetes/commit/a38b9363ec3f0c10bb6e6ebc47d731d6c88ae03f#r126178773

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 2, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/kube-proxy sig/network Categorizes an issue or PR as relevant to SIG Network. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 2, 2023
@aojea
Copy link
Member

aojea commented Sep 2, 2023

You have to adapt the unit test , check if this can serve as guide #120050

@pegasas pegasas changed the title Improve logging on kube-proxy exit [DRAFT]Improve logging on kube-proxy exit Sep 3, 2023
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 3, 2023
@aojea
Copy link
Member

aojea commented Sep 3, 2023

/assign @aojea @bowei @pohly

This looks good, waiting for the test results

@pegasas pegasas changed the title [DRAFT]Improve logging on kube-proxy exit Improve logging on kube-proxy exit Sep 3, 2023
@aojea
Copy link
Member

aojea commented Sep 3, 2023

/approve

Waiting for @pohly or @bowei for lgtm

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, pegasas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 3, 2023
pohly
pohly reviewed Sep 6, 2023
View reviewed changes
Copy link
Contributor

@pohly pohly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but before I /lgtm let me mention another thought - perhaps you'll want to fix this in this PR.

Would it make sense to support writing a termination message in kube-proxy?

@aojea
Copy link
Member

aojea commented Sep 6, 2023

Would it make sense to support writing a termination message in kube-proxy?

that is a container field, isn't it?
we can't assume kube-proxy runs in a pod and we don't control kube-proxy deployments, or am I missing something?

yuzhiquan
yuzhiquan reviewed Sep 7, 2023
View reviewed changes
pkg/proxy/node.go
@@ -57,7 +57,7 @@ func (n *NodePodCIDRHandler) OnNodeAdd(node *v1.Node) {
if !reflect.DeepEqual(n.podCIDRs, podCIDRs) {
klog.ErrorS(nil, "Using NodeCIDR LocalDetector mode, current PodCIDRs are different than previous PodCIDRs, restarting",
"node", klog.KObj(node), "newPodCIDRs", podCIDRs, "oldPodCIDRs", n.podCIDRs)
panic("Current Node PodCIDRs are different than previous PodCIDRs, restarting")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will lost the exit message

Suggested change
klog.FlushAndExit(klog.ExitFlushTimeout, 1)
klog.InfoS("Current Node PodCIDRs are different than previous PodCIDRs, restarting")
klog.FlushAndExit(klog.ExitFlushTimeout, 1)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure.

Copy link
Contributor Author

@pegasas pegasas Sep 7, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems klog.ErrorS log this in the error level,
need we log it another time?

@yuzhiquan

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ops, sorry, miss that.

@pegasas
Copy link
Contributor Author

pegasas commented Sep 7, 2023

we can't assume kube-proxy runs in a pod and we don't control kube-proxy deployments, or am I missing something?

What you can do is add an env variable or command line flag for the termination message file. Then someone who deploys kube-proxy in a pod can enable that and the reason why kube-proxy terminated will be visible in the pod status. That's better than writing to the log file.

Hi, @pohly ,

Yes, for sure.

It seems this is another proposal.

When we host kube-proxy in a pod, we should write an error reason related log into /dev/termination-log,
then kubectl could get more meta messages about it.

I would like to open another thread for tracking.

@pohly
Copy link
Contributor

pohly commented Sep 7, 2023

Doing this in a follow-up makes sense.

@pegasas pegasas requested a review from pohly September 8, 2023 02:29
@yuzhiquan
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 9, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 52afa76a4ab9648251a16b4e525d39f6ab155e0b

@pacoxu
Copy link
Member

pacoxu commented Sep 10, 2023

Does this need a release note?

@aojea
Copy link
Member

aojea commented Sep 10, 2023

/release-note-edit

kube-proxy don't panic on exit when the Node object changes its PodCIDR

@k8s-ci-robot
Copy link
Contributor

@aojea: /release-note-edit must be used with a single release note block.

In response to this:

/release-note-edit

kube-proxy don't panic on exit when the Node object changes its PodCIDR

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@aojea
Copy link
Member

aojea commented Sep 10, 2023

/release-note-edit

kube-proxy don't panic on exit when the Node object changes its PodCIDR

@pegasas please update the release note block in the description

@pegasas
Copy link
Contributor Author

pegasas commented Sep 10, 2023

/release-note-edit

kube-proxy don't panic on exit when the Node object changes its PodCIDR

@pegasas please update the release note block in the description

@aojea

Sure.
note block updated.

@aojea
Copy link
Member

aojea commented Sep 10, 2023 •
edited

you need to put a block of text like this

 ```release-note
 kube-proxy don't panic on exit when the Node object changes its PodCIDR

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Sep 10, 2023
@pegasas
Copy link
Contributor Author

pegasas commented Sep 10, 2023

you need to put a block of text like this

 ```release-note
 kube-proxy don't panic on exit when the Node object changes its PodCIDR

Thanks!
Corrected.

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 10, 2023
@aojea
Copy link
Member

aojea commented Sep 10, 2023

FYI you don't need to rebase, the CI tooling does it for you ;)

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 10, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: e1e8be1db8296425a2e52a4a8609160de0f2cf90

@k8s-ci-robot k8s-ci-robot merged commit 0ee315b into kubernetes:master Sep 10, 2023
14 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Sep 10, 2023
@pegasas pegasas deleted the proxy branch September 10, 2023 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@khenidak khenidak Awaiting requested review from khenidak

@robscott robscott Awaiting requested review from robscott

@pohly pohly Awaiting requested review from pohly

@aojea aojea Awaiting requested review from aojea

@bowei bowei Awaiting requested review from bowei

@yuzhiquan yuzhiquan Awaiting requested review from yuzhiquan

Assignees

@pohly pohly

@bowei bowei

@yuzhiquan yuzhiquan

@aojea aojea

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kube-proxy cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/network Categorizes an issue or PR as relevant to SIG Network. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.29
Development

Successfully merging this pull request may close these issues.

Improve logging on kube-proxy exit
7 participants
@pegasas @k8s-ci-robot @aojea @pohly @yuzhiquan @pacoxu @bowei