Ignore AWS NodeWithImpairedVolumes taint

[johanneswuerbach]

The NodeWithImpairedVolumes taint is applied to node on AWS when a volume is stuck in attaching state for too long.

The taint was introduced in k8s a while ago kubernetes/kubernetes#55558

Add it to the ignored node condition taints.

[johanneswuerbach]

/assign @losipiuk

[johanneswuerbach]

/area provider/aws

[johanneswuerbach]

Sorry, maybe @Jeffwan would be a better assignee as this is AWS specific.

[Jeffwan]

@johanneswuerbach

Code changes look good to me

In your case, what's the root cause for the volume takes that long to attach?

[Jeffwan]

@johanneswuerbach Code changes look good to me

/lgtm

In your case, what's the root cause for the volume takes that long to attach?

[johanneswuerbach]

Not sure, it looks like an issue in AWS, but it might also be k8s related. We currently run on 1.16 provisioned using kops and I requested a back port of a fix, which might be related kubernetes/kubernetes#89894

[Jeffwan]

@johanneswuerbach em, good to know, I will ask my team member to track this change and try to get it approved

[johanneswuerbach]

@Jeffwan any update on this?

[johanneswuerbach]

/assign @aleksandra-malinowska

[mwielgus]

@Jeffwan ping :)

[Jeffwan]

/lgtm

needs someone to approve this change. @mwielgus

[edsonmarquezani]

I've seen a side effect of it.

I have an instance group whose applications are shut down at a certain scheduled. When there's a node tainted like this, this specific node won't be terminated and will be left as the only one for that instance group. The day after, when PODs are up again, the autoscaler WON'T scale nodes up because of this taint, showing a message like this.

I0622 12:35:50.236346 1 utils.go:196] Pod wololo-67ccdcb765-xd7dq can't be scheduled
on spot-nodes, predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put
forno/wololo-67ccdcb765-xd7dq on template-node-for-spot-nodes-176367186976534907,
reason: node(s) had taints that the pod didn't tolerate

If I scale one node manually on AWS or terminate the tainted node, things get back to work again.

[johanneswuerbach]

That is actually what the PR is supposed to solve and solved for us.

As a workaround adding --ignore-taint=NodeWithImpairedVolumes should fix your issue also with a the current autoscaler.

[edsonmarquezani]

@joshbranham Thanks for the advice!

[johanneswuerbach]

@mwielgus @Jeffwan anything missing here?

[MaciekPytel]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MaciekPytel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS [MaciekPytel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment