add node label selector to Terminator #625
Conversation
| Expect(drainedNodes).To(BeEmpty()) | ||
| }) | ||
|
|
||
| It("does not delete the message from the SQS queue", func() { |
There was a problem hiding this comment.
How long do messages stay on the SQS queue? It seems like there could be a failure mode where NTH spends all its time re-processing messages on the queue (because, e.g., the labels didn't match) and isn't able to respond to new messages in time for k8s to gracefully handle a node termination. That's probably uncommon, but it is basically what mechanical-fish described in #494.
Is that a concern with this way of handling node labels? It seems like the philosophy of the terminator is "I only respond to requests that match my parameters; anything else I will leave for someone else to handle". So it may not be possible to prevent that situation. Curious what your thoughts are.
There was a problem hiding this comment.
Queues have a message retention period setting. A dead-letter queue can also be configured.
Leaving the message in the queue gives another Terminator instance a chance to match and handle it.
Issue #, if available:
#556 Initial NTH v2 design proposal
Description of changes:
Add
matchLabelsproperty to the Terminator spec.NTH will take no action when an SQS message targets a node that does not have the requisite labels and values.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.