feat: Add support for Auto Scaling Group Instance Refresh for self-managed worker groups

[bashims]

PR o'clock

Description

Add support for the instance_refresh feature of the AWS provisioner added in hashicorp/terraform-provider-aws#16678

This PR resolves #929

See:
https://aws.amazon.com/blogs/compute/introducing-instance-refresh-for-ec2-auto-scaling/

Checklist

• CI tests are passing
• README.md has been updated after any changes to variables and outputs. See https://github.com/terraform-aws-modules/terraform-aws-eks/#doc-generation

[stevehipwell]

@bashims does the phased deployments anouncement make any difference to this PR?

It looks like you haven't configured the ASGs to work with the aws-node-termination-handler (queue processor mode, chart)? I would have expected this to be a requirement as you wouldn't want a node to suddenly terminate with no warning.

[bashims]

@bashims does the phased deployments anouncement make any difference to this PR?

Good question! I am not too sure if the checkpoint feature is available in the AWS provider. For now it seems to support the original instance refresh feature.

It looks like you haven't configured the ASGs to work with the aws-node-termination-handler (queue processor mode, chart)? I would have expected this to be a requirement as you wouldn't want a node to suddenly terminate with no warning.

Good point. I can add something to the docs to mention that one should deploy the aws-node-termination-handler chart when enabling instance refresh. The eks TF module does not provide support for deploying charts etc, so I doubt it would be a good fit here.

[stevehipwell]

@bashims I wasn't refering to actually installing the chart but you'll need to set up a SQS queue for the events and then provide the chart installation instructions to work with instance refresh (specifically controller not daemonset mode).

[bashims]

@bashims I wasn't refering to actually installing the chart but you'll need to set up a SQS queue for the events and then provide the chart installation instructions to work with instance refresh (specifically controller not daemonset mode).

@stevehipwell Awesome feedback. I was not aware. I will update this PR to include this feature.

[Erokos]

Hi, what's the status with this PR?

[Erokos]

Hi,
one other question regarding the work in this PR is, when this gets merges and used, will the "asg_recreate_on_change" param still be used/necessary?

[barryib]

Hi,
one other question regarding the work in this PR is, when this gets merges and used, will the "asg_recreate_on_change" param still be used/necessary?

Good question. I don't think so. I think with this addition we can drop the asg_recreate_on_change and random pet names.

@bashims @stevehipwell I think we should add in docs that using instance refresh will recreate your nodes, but it doesn't drains them. To do that, users should install https://github.com/aws/aws-node-termination-handler

Related to:

• How to handle worker upgrade automatically ? #462

[stevehipwell]

@bashims @barryib I assume that this PR will add the SQS queue that aws-node-termination-handler needs. So if instance refresh is turned on you need to install aws-node-termination-handler for draining but everything else works?

[Erokos]

I wholeheartedly agree. From reading the issues, I've come across references on random_pet usage and from the docs it isn't highlighted that using asg_recreate_on_change doesn't drain nodes and in my opinion it should say so. To some it may be obvious but a warning never hurt nobody 🙂. I could be wrong but without having to manage the draining yourself, by having a shell script within a null resource (with a remote-exec) or something similar, I don't see how else it could be done.

[barryib]

@bashims @barryib I assume that this PR will add the SQS queue that aws-node-termination-handler needs. So if instance refresh is turned on you need to install aws-node-termination-handler for draining but everything else works?

Humm... But users should handle aws-node-terminination-handler installation with all requirements themselves. In this module, we'll only support changes in worker ASG like:

• instance refresh configuration
• lifecycle hooks configuration

Everything else should be handled outside of this module. We can write docs for that. If someone willing to write a module to address aws-node-termination-handler installation, we can probably include that link in docs.

[stevehipwell]

@barryib I think the SQS linking is part of the ASG configuration, so at the minimum that would need to be passed into the module so the aws-node-termination-handler can be linked up correctly.

[barryib]

@barryib I think the SQS linking is part of the ASG configuration, so at the minimum that would need to be passed into the module so the aws-node-termination-handler can be linked up correctly.

Yep. It'll be part of the lifecycle hook configuration https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/local.tf#L37

[bashims]

Hi,
one other question regarding the work in this PR is, when this gets merges and used, will the "asg_recreate_on_change" param still be used/necessary?

Good question. I don't think so. I think with this addition we can drop the asg_recreate_on_change and random pet names.

@bashims @stevehipwell I think we should add in docs that using instance refresh will recreate your nodes, but it doesn't drains them. To do that, users should install https://github.com/aws/aws-node-termination-handler

Related to:

* #462

I can take care of that in this PR.

[barryib]

I can take care of that in this PR.

Thanks @bashims. Can you please:

• Update docs to highlight the need of aws-node-termination handler for draining
• Open a new PR (if you have time) to drop asg_recreate_on_change and random_pets in for LT and LC

[bashims]

I can take care of that in this PR.

Thanks @bashims. Can you please:

* Update docs to highlight the need of aws-node-termination handler for draining

* Open a new PR (if you have time) to drop `asg_recreate_on_change` and random_pets in for LT and LC

Absolutely.

[bashims]

@bashims @barryib I assume that this PR will add the SQS queue that aws-node-termination-handler needs. So if instance refresh is turned on you need to install aws-node-termination-handler for draining but everything else works?

Humm... But users should handle aws-node-terminination-handler installation with all requirements themselves. In this module, we'll only support changes in worker ASG like:

* instance refresh configuration

* lifecycle hooks configuration

Everything else should be handled outside of this module. We can write docs for that. If someone willing to write a module to address aws-node-termination-handler installation, we can probably include that link in docs.

[bashims]

@barryib I think this PR is nearly ready. I am working on complete example that includes necessary setup for graceful node draining with instance refresh. I should have this done soon.

[bashims]

I think that this PR is ready to go. The complete example works perfectly.

Just waiting on workflow approval, and final review. Once that is done I will open a PR for dropping asg_recreate_on_change and random pet names.

[barryib]

@bashims I just opened #1360 to address the random pets removal.

[barryib]

@bashims you need to update your branch and resolve conflicts.

[bashims]

@barryib are you okay if I squash and rebase off of master?

[barryib]

@barryib are you okay if I squash and rebase off of master?

Yep.

[barryib]

Thanks @bashims for your contributions.

[barryib]

This now shipped in v16.0.0

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.