Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop flooding the kube-proxy logs on dual-stack because of IPFamily #88934

Merged
merged 2 commits into from Mar 10, 2020

Conversation

@aojea
Copy link
Member

aojea commented Mar 6, 2020

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:

/kind api-change

/kind bug
/kind cleanup

/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

The kube-proxy metaproxier implementations tries to get the IPFamily
from the endpoints, but if the endpoints doesn't contains an IP
address it logs a Warning.

This causes that services without endpoints keep flooding the logs
with warnings.

We can log this errors with a level of Verbosity of 4 instead of a Warning

Which issue(s) this PR fixes:

Fixes #88784

Special notes for your reviewer:

It also adds unit tests to the metaproxier package.

Does this PR introduce a user-facing change?:

kube-proxy: on dual-stack mode, if it is not able to get the IP Family of an endpoint, logs it with level InfoV(4) instead of Warning, avoiding flooding the logs for endpoints without addresses

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 6, 2020

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Mar 6, 2020

@aojea: GitHub didn't allow me to request PR reviews from the following users: uablrek.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/assign @thockin @robscott
/cc @uablrek

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added sig/network and removed needs-sig labels Mar 6, 2020
@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 6, 2020

/sig network
/priority important-soon

The kube-proxy metaproxier implementations tries to get the IPFamily
from the endpoints, but if the endpoints doesn't contains an IP
address it logs a Warning.

This causes that services without endpoints keep flooding the logs
with warnings.

We log this errors with a level of Verbosity of 4 instead of a Warning
@aojea aojea force-pushed the aojea:endpointnolog branch from 1ed4a9d to df58c04 Mar 7, 2020
@@ -103,7 +103,7 @@ func (proxier *metaProxier) OnServiceSynced() {
func (proxier *metaProxier) OnEndpointsAdd(endpoints *v1.Endpoints) {
ipFamily, err := endpointsIPFamily(endpoints)
if err != nil {
klog.Warningf("failed to add endpoints %s/%s with error %v", endpoints.ObjectMeta.Namespace, endpoints.ObjectMeta.Name, err)
klog.V(4).Infof("failed to add endpoints %s/%s with error %v", endpoints.ObjectMeta.Namespace, endpoints.ObjectMeta.Name, err)

This comment has been minimized.

Copy link
@aojea

aojea Mar 7, 2020

Author Member

Maybe we should use V(1)?
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md

klog.V(1) - A reasonable default log level if you don't want verbosity.

Information about config (listening on X, watching Y)
Errors that repeat frequently that relate to conditions that can be corrected (pod detected as unhealthy)

This comment has been minimized.

Copy link
@dcbw

dcbw Mar 9, 2020

Member

@aojea I'm fine with V(4). endpointsIPFamily() returns errors when (a) there are no subsets, (b) there are no subset addresses, (c) the first subset's ready IP is empty.

If we look at validation, then (a) is valid eg endpoints don't need subsets, (b) is valid because there may be not-ready addresses but no ready addresses, and (c) should have been rejected by validation due to validateEndpointAddress() and validation.IsValidIP().

eg, none of these things that endpointsIPFamily() returns should be logged with a Warningf() because they are either valid configurations of endpoints/subsets, or are cases that shouldn't be hit becuase they would have been rejected by validation.

@dcbw

This comment has been minimized.

Copy link
Member

dcbw commented Mar 9, 2020

/lgtm
/approve

@dcbw

This comment has been minimized.

Copy link
Member

dcbw commented Mar 9, 2020

/milestone v1.18

@k8s-ci-robot k8s-ci-robot added this to the v1.18 milestone Mar 9, 2020
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Mar 9, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, dcbw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Mar 9, 2020

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 9, 2020

/retest

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Mar 10, 2020

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 10, 2020

/retest
TestStressingCascadingDeletion

[sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a pod. [Conformance] expand_less

@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 10, 2020

/retest
🤔 TestStressingCascadingDeletion again?

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Mar 10, 2020

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 10, 2020

/hold

this test failed at least 3 times in this PR
TestStressingCascadingDeletion

@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 10, 2020

/test pull-kubernetes-integration

@aojea

This comment has been minimized.

Copy link
Member Author

aojea commented Mar 10, 2020

/hold cancel
failures seems unrelated

@k8s-ci-robot k8s-ci-robot merged commit 0ec85a1 into kubernetes:master Mar 10, 2020
16 of 17 checks passed
16 of 17 checks passed
tide Not mergeable. Retesting: pull-kubernetes-verify
Details
cla/linuxfoundation aojea authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-kind-ga-only-parallel Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kind Job succeeded.
Details
pull-kubernetes-e2e-kind-ipv6 Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.