New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail container start if its requested device plugin resource is unknown. #67145

Merged
merged 1 commit into from Aug 21, 2018

Conversation

@jiayingz
Member

jiayingz commented Aug 8, 2018

With the change, Kubelet device manager now checks whether it has cached option state for the requested device plugin resource to make sure the resource is in ready state when we start the container.

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #67107

Special notes for your reviewer:

Release note:

Fail container start if its requested device plugin resource hasn't registered after Kubelet restart.
@jiayingz

This comment has been minimized.

Show comment
Hide comment
@jiayingz

jiayingz Aug 8, 2018

Member

/sig node

Member

jiayingz commented Aug 8, 2018

/sig node

@jiayingz

This comment has been minimized.

Show comment
Hide comment
@jiayingz
Member

jiayingz commented Aug 8, 2018

@RenaudWasTaken

This comment has been minimized.

Show comment
Hide comment
@RenaudWasTaken

RenaudWasTaken Aug 8, 2018

Member

Are failed container "retried" or is this up to another abstraction?

Member

RenaudWasTaken commented Aug 8, 2018

Are failed container "retried" or is this up to another abstraction?

@jiayingz

This comment has been minimized.

Show comment
Hide comment
@jiayingz

jiayingz Aug 8, 2018

Member

Kubelet will retry with exponential backoff latency.

Member

jiayingz commented Aug 8, 2018

Kubelet will retry with exponential backoff latency.

Fail container start if its requested device plugin resource doesn't
have cached option state to make sure the device plugin resource is
in ready state when we start the container.
@vikaschoudhary16

This comment has been minimized.

Show comment
Hide comment
@vikaschoudhary16
Member

vikaschoudhary16 commented Aug 21, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Aug 21, 2018

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Aug 21, 2018

Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiayingz, vikaschoudhary16

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Contributor

k8s-ci-robot commented Aug 21, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiayingz, vikaschoudhary16

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Aug 21, 2018

Contributor

/test all

Tests are more than 96 hours old. Re-running tests.

Contributor

k8s-merge-robot commented Aug 21, 2018

/test all

Tests are more than 96 hours old. Re-running tests.

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Aug 21, 2018

Contributor

/test all [submit-queue is verifying that this PR is safe to merge]

Contributor

k8s-merge-robot commented Aug 21, 2018

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Aug 21, 2018

Contributor

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

Contributor

k8s-merge-robot commented Aug 21, 2018

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-merge-robot k8s-merge-robot merged commit d017beb into kubernetes:master Aug 21, 2018

16 of 18 checks passed

Submit Queue Required Github CI test is not green: pull-kubernetes-kubemark-e2e-gce-big
Details
pull-kubernetes-local-e2e-containerized Job triggered.
Details
cla/linuxfoundation jiayingz authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
@dashpole

This comment has been minimized.

Show comment
Hide comment
@dashpole
Contributor

dashpole commented Aug 22, 2018

@jiayingz

This comment has been minimized.

Show comment
Hide comment
@jiayingz

jiayingz Aug 23, 2018

Member

Ah right. I need to update the device plugin e2e_node test to cope with this behavior change. I will send a PR to fix that. @dashpole thanks a lot for the notification!

Member

jiayingz commented Aug 23, 2018

Ah right. I need to update the device plugin e2e_node test to cope with this behavior change. I will send a PR to fix that. @dashpole thanks a lot for the notification!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment