BUG 1804738: Ensure DeleteNodes doesn't delete a node twice

[JoelSpeed]

I have seen the autoscaler call DeleteNodes several times while a node it is trying to delete has not been fully deleted (finalizer still present). Currently, it does not check if the node's Machine has already been deleted and blindly reduces the size of the node group by 1 on every call to DeleteNodes

This PR adds a check, so that if a node's Machine already has a deletion timestamp, we do not decrease the size of the node group any further

I've added a test that calls the DeleteNodes twice and expects the node group size to not decrease on the second call, which passes with the fix, and fails without

[enxebre]

cc @bison @frobware

[enxebre]

/hold
until we get a bz
/cherry-pick release-4.4
/cherry-pick release-4.3

[openshift-cherrypick-robot]

@enxebre: once the present PR merges, I will cherry-pick it on top of release-4.4 in a new PR and assign it to you.

In response to this:

/hold
until we get a bz
/cherry-pick release-4.4
/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@JoelSpeed: This pull request references Bugzilla bug 1804738, which is invalid:

• expected the bug to target the "4.5.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

BUG 1804738: Ensure DeleteNodes doesn't delete a node twice

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alexander-demicev]

/bugzilla refresh

[openshift-ci-robot]

@alexander-demichev: This pull request references Bugzilla bug 1804738, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[enxebre]

/hold cancel

[enxebre]

lgtm PTAL @frobware, would like your blessing here

[enxebre]

/lgtm

[enxebre]

thanks! @JoelSpeed please make sure to follow up with the cherry-picks

[JoelSpeed]

/retest

[openshift-ci-robot]

@JoelSpeed: All pull requests linked via external trackers have merged. Bugzilla bug 1804738 has been moved to the MODIFIED state.

In response to this:

BUG 1804738: Ensure DeleteNodes doesn't delete a node twice

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-cherrypick-robot]

@enxebre: new pull request created: #128

In response to this:

/hold
until we get a bz
/cherry-pick release-4.4
/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[JoelSpeed]

/cherry-pick release-4.3

[openshift-cherrypick-robot]

@JoelSpeed: new pull request created: #129

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[frobware]

lgtm PTAL @frobware, would like your blessing here

Yep, sorry (github outage yesterday). I did have one minor nit, but not important now.