Fix drain logic when skipNodesWithCustomControllerPods=false, set NodeDeleteOptions correctly

[BigDarkClown]

What type of PR is this?

/kind fix

What this PR does / why we need it:

In drain logic the isDaemonSet variable is set outside of pod loop. It is not correctly changed when skipNodesWithCustomControllerPods=false, so if there is at least one DS pod on the node it can be drained. This is also non-deterministic as the slice of pods is passed in random order.

Additionally the NodeDeleteOptions were not set in all appropriate places. Because of this some parts of the logic like empty candidate sorting always worked on skipNodesWithCustomControllerPods=false.

Original change: #5507

[k8s-ci-robot]

@BigDarkClown: The label(s) kind/fix cannot be applied, because the repository doesn't have them.

In response to this:

What type of PR is this?

/kind fix

What this PR does / why we need it:

In drain logic the isDaemonSet variable is set outside of pod loop. It is not correctly changed when skipNodesWithCustomControllerPods=false, so if there is at least one DS pod on the node it can be drained. This is also non-deterministic as the slice of pods is passed in random order.

Additionally the NodeDeleteOptions were not set in all appropriate places. Because of this some parts of the logic like empty candidate sorting always worked on skipNodesWithCustomControllerPods=false.

Original change: #5507

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BigDarkClown]

/assign x13n

[x13n]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BigDarkClown, x13n

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS [x13n]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment