Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove wait.Until for running Kubelet Bootstrap #89055

Merged
merged 1 commit into from Mar 13, 2020

Conversation

@tedyu
Copy link
Contributor

tedyu commented Mar 11, 2020

What type of PR is this?
/kind bug

What this PR does / why we need it:
As @liggitt outlined in #88779 (comment) , there are two options in preventing multiple parallel Run goroutines.

This PR adopts option #1.

Which issue(s) this PR fixes:
Fixes #88779

Fixed an issue that could cause the kubelet to incorrectly run concurrent pod reconciliation loops and crash.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


Signed-off-by: Ted Yu <yuzhihong@gmail.com>
@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 11, 2020

/priority important-soon

@sjenning

This comment has been minimized.

Copy link
Contributor

sjenning commented Mar 11, 2020

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 11, 2020

/test pull-kubernetes-e2e-kind

@sjenning

This comment has been minimized.

Copy link
Contributor

sjenning commented Mar 11, 2020

/cc @roycaihw

@k8s-ci-robot k8s-ci-robot requested a review from roycaihw Mar 11, 2020
@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 12, 2020

@derekwaynecarr
Can you review this ?

@roycaihw

This comment has been minimized.

Copy link
Member

roycaihw commented Mar 13, 2020

/lgtm

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Mar 13, 2020

Given the discussion in #88779 (comment) and the fact that invoking k.Run() twice in the same process is clearly incorrect, I'd like to merge this and pick it to 1.15/1.16/1.17.

Would like an ack from kubelet owners on that approach.

@derekwaynecarr

This comment has been minimized.

Copy link
Member

derekwaynecarr commented Mar 13, 2020

/lgtm
/approve

we can do pick once merged

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Mar 13, 2020

/milestone v1.18
resolves kubelet-crashing bug, candidate for picking to all release branches

@k8s-ci-robot k8s-ci-robot added this to the v1.18 milestone Mar 13, 2020
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Mar 13, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, tedyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Mar 13, 2020

flake #86318
/retest

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 13, 2020

Errors from staticcheck:
vendor/k8s.io/csi-translation-lib/plugins/gce_pd.go:51:2: const nodeIDFmt is unused (U1000)

Not related to my PR

The constant is used by:
staging/src/k8s.io/csi-translation-lib/plugins/gce_pd_test.go

@rphillips

This comment has been minimized.

Copy link
Member

rphillips commented Mar 13, 2020

/retest

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 13, 2020

/test pull-kubernetes-e2e-gce

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 13, 2020

Not sure what happened for the pull-kubernetes-verify job:

{"component":"entrypoint","file":"prow/entrypoint/run.go:245","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 15s grace period","time":"2020-03-13T21:20:58Z"}
@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 13, 2020

/test pull-kubernetes-verify

@k8s-ci-robot k8s-ci-robot merged commit 2b63c09 into kubernetes:master Mar 13, 2020
16 of 17 checks passed
16 of 17 checks passed
tide Not mergeable. Retesting: pull-kubernetes-verify
Details
cla/linuxfoundation tedyu authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-kind-ga-only-parallel Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kind Job succeeded.
Details
pull-kubernetes-e2e-kind-ipv6 Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 19, 2020

@derekwaynecarr @liggitt
Should I open backport PR for 1.17 branch ?

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Mar 20, 2020

yes, please. 1.15, 1.16, 1.17.

@tedyu

This comment has been minimized.

Copy link
Contributor Author

tedyu commented Mar 20, 2020

@liggitt
#89278 through #89280 have been created for the backports.

k8s-ci-robot added a commit that referenced this pull request Apr 4, 2020
…upstream-release-1.16

Automated cherry pick of #89055: Remove wait.Until for running Kubelet Bootstrap
k8s-ci-robot added a commit that referenced this pull request Apr 4, 2020
…upstream-release-1.15

Automated cherry pick of #89055: Remove wait.Until for running Kubelet Bootstrap
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.