OCPBUGS-8287: [release-4.10] UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy device exists

[swatisehgal]

What this PR does / why we need it:

This is a targeted cherry pick containing only the core fix from kubernetes#116376.

Rationale for the backporting the bugfix
During the allocation phase, we need to ensure that the resources requested by the pod should only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated.
If these conditions are not satisfied (which can happen in case of node reboot/kubelet restart because there is no way to control the pod restart order), the pod would fail with UnexpectedAdmissionError error.

For more details on the rationale, refer to the comment here: kubernetes#109595 (comment)

Additional Notes for the reviewers:
Validation of the change can be performed using openshift/origin#27902.

[openshift-ci]

@swatisehgal: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.10] UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy device exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@swatisehgal: the contents of this pull request could be automatically validated.

The following commits are valid:

• 0f1feac|UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy devices exist: the upstream PR kubernetes/kubernetes#116376 has merged

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci]

@swatisehgal: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

OCPBUGS-2180: [release-4.10] UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy device exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@swatisehgal: This pull request references Jira Issue OCPBUGS-2180, which is invalid:

• expected the bug to target the "4.10.z" version, but it targets "4.13.0" instead
• expected Jira Issue OCPBUGS-2180 to depend on a bug targeting a version in 4.11.0, 4.11.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What this PR does / why we need it:

This is a targeted cherry pick containing only the core fix from https://github.com/kubernetes/kubernetes/pull//116337.

Rationale for the backporting the bugfix
During the allocation phase, we need to ensure that the eesources requested by the pod should only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated.
If these conditions are not satisfied (which can happen in case of node reboot/kubelet restart because there is no way to control the pod restart order), the pod would fail with UnexpectedAdmissionError error.

For more details on the rationale, refer to the comment here: kubernetes#109595 (comment)

Additional Notes for the reviewers:
Validation of the change can be performed using openshift/origin#27902.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@swatisehgal: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.10] UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy device exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@swatisehgal: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

In response to this:

What this PR does / why we need it:

This is a targeted cherry pick containing only the core fix from https://github.com/kubernetes/kubernetes/pull//116337.

Rationale for the backporting the bugfix
During the allocation phase, we need to ensure that the eesources requested by the pod should only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated.
If these conditions are not satisfied (which can happen in case of node reboot/kubelet restart because there is no way to control the pod restart order), the pod would fail with UnexpectedAdmissionError error.

For more details on the rationale, refer to the comment here: kubernetes#109595 (comment)

Additional Notes for the reviewers:
Validation of the change can be performed using openshift/origin#27902.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@swatisehgal: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.10] UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy device exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@swatisehgal: the contents of this pull request could be automatically validated.

The following commits are valid:

• df4c0e9|UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy devices exist: the upstream PR kubernetes/kubernetes#116376 has merged

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@swatisehgal: the contents of this pull request could be automatically validated.

The following commits are valid:

• 613c8d1|UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy devices exist: the upstream PR kubernetes/kubernetes#116376 has merged

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[swatisehgal]

/test e2e-aws-cgroupsv2

[swatisehgal]

/test e2e-gcp

[openshift-ci]

@swatisehgal: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.10] UPSTREAM: 116376: node: device-mgr: Handle recovery by checking if healthy device exists

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[swatisehgal]

/test k8s-e2e-aws

[openshift-ci]

@swatisehgal: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[swatisehgal]

/jira refresh

[openshift-ci-robot]

@swatisehgal: This pull request references Jira Issue OCPBUGS-8287, which is invalid:

• expected dependent Jira Issue OCPBUGS-14140 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[swatisehgal]

/cc @soltysh

[soltysh]

/lgtm
/approve
/label backport-risk-assessed

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: soltysh, swatisehgal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/DOWNSTREAM_OWNERS [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[swatisehgal]

/jira refresh

[openshift-ci-robot]

@swatisehgal: This pull request references Jira Issue OCPBUGS-8287, which is invalid:

• expected dependent Jira Issue OCPBUGS-14140 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-ci-robot]

@swatisehgal: This pull request references Jira Issue OCPBUGS-8287, which is invalid:

• expected dependent Jira Issue OCPBUGS-14140 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

What this PR does / why we need it:

This is a targeted cherry pick containing only the core fix from kubernetes#116376.

Rationale for the backporting the bugfix
During the allocation phase, we need to ensure that the resources requested by the pod should only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated.
If these conditions are not satisfied (which can happen in case of node reboot/kubelet restart because there is no way to control the pod restart order), the pod would fail with UnexpectedAdmissionError error.

For more details on the rationale, refer to the comment here: kubernetes#109595 (comment)

Additional Notes for the reviewers:
Validation of the change can be performed using openshift/origin#27902.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[swatisehgal]

/close
We longer plan to backport the fix to 4.10 branch.

[openshift-ci]

@swatisehgal: Closed this PR.

In response to this:

/close
We longer plan to backport the fix to 4.10 branch.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@swatisehgal: This pull request references Jira Issue OCPBUGS-8287. The bug has been updated to no longer refer to the pull request using the external bug tracker.

In response to this:

What this PR does / why we need it:

This is a targeted cherry pick containing only the core fix from kubernetes#116376.

Rationale for the backporting the bugfix
During the allocation phase, we need to ensure that the resources requested by the pod should only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated.
If these conditions are not satisfied (which can happen in case of node reboot/kubelet restart because there is no way to control the pod restart order), the pod would fail with UnexpectedAdmissionError error.

For more details on the rationale, refer to the comment here: kubernetes#109595 (comment)

Additional Notes for the reviewers:
Validation of the change can be performed using openshift/origin#27902.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.