Referencing the drainTimeout value in the NodeDrainer daemonset

[HarryStericker]

In a previous PR (#1722) I had attempted to stop node drainer scheduling on cordoned nodes once the timeout had elapsed (300s/5m). This is beneficial to me as otherwise nodeDrainer tries to redeploy itself onto the node after the 5 minutes have elapsed, and goes into a restart loop as kubelet is preventing things from being scheduled. When this happens our alerts go off.

This proved more difficult than planned, and I can not think of any other way to do so than placing a taint on all nodes that the daemonset does not have a toleration for. This is IMO not feasible and way too heavyweight.

We already have drainTimeout specified here in cluster.yml so it makes sense to use this value in the drain command also. This way I can increase the timeout allowing the node to drain and terminate within the allotted time.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign danielfm
You can assign the PR to them by writing /assign @danielfm in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-io]

Codecov Report

Merging #1731 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1731   +/-   ##
=======================================
  Coverage   25.14%   25.14%           
=======================================
  Files          98       98           
  Lines        5027     5027           
=======================================
  Hits         1264     1264           
  Misses       3621     3621           
  Partials      142      142

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fe0077...484b691. Read the comment docs.

[dominicgunn]

/lgtm

[davidmccormick]

Many thanks for your contribution! 🙏