feat!: Replace the use of aws-auth configmap with EKS cluster access entry

[bryantbiggs]

Description

List of backwards incompatible changes

• Minium supported AWS provider version increased to v5.34
• Minimum supported Terraform version increased to v1.3 to support Terraform state moved blocks as well as other advanced features
• The resolve_conflicts argument within the cluster_addons configuration has been replaced with resolve_conflicts_on_create and resolve_conflicts_on_delete now that resolve_conflicts is deprecated
• The default/fallback value for the preserve argument of cluster_addonsis now set to true. This has shown to be useful for users deprovisioning clusters while avoiding the situation where the CNI is deleted too early and causes resources to be left orphaned resulting in conflicts.
• The Karpenter sub-module's use of the irsa naming convention has been removed, along with an update to the Karpenter controller IAM policy to align with Karpenter's v1beta1/v0.32 changes. Instead of referring to the role as irsa or pod_identity, its simply just an IAM role used by the Karpenter controller and there is support for use with either IRSA and/or Pod Identity (default) at this time
• The aws-auth ConfigMap resources have been moved to a standalone sub-module. This removes the Kubernetes provider requirement from the main module and allows for the aws-auth ConfigMap to be managed independently of the main module. This sub-module will be removed entirely in the next major release.
• Support for cluster access management has been added with the default authentication mode set as API_AND_CONFIG_MAP. This is a one way change if applied; if you wish to use CONFIG_MAP, you will need to set authentication_mode = "CONFIG_MAP" explicitly when upgrading.
• Karpenter EventBridge rule key spot_interrupt updated to correct mis-spelling (was spot_interupt). This will cause the rule to be replaced

Additional changes

Added

• A module tag has been added to the cluster control plane
• Support for cluster access entries. The bootstrap_cluster_creator_admin_permissions setting on the control plane has been hardcoded to false since this operation is a one time operation only at cluster creation per the EKS API. Instead, users can enable/disable enable_cluster_creator_admin_permissions at any time to achieve the same functionality. This takes the identity that Terraform is using to make API calls and maps it into a cluster admin via an access entry. For users on existing clusters, you will need to remove the default cluster administrator that was created by EKS prior to the cluster access entry APIs - see the section Removing the default cluster administrator for more details.
• Support for specifying the CloudWatch log group class (standard or infrequent access)
• Native support for Windows based managed nodegroups similar to AL2 and Bottlerocket
• Self-managed nodegroups now support instance_maintenance_policy and have added max_healthy_percentage, scale_in_protected_instances, and standby_instances arguments to the instance_refresh.preferences block

Modified

• For sts:AssumeRole permissions by services, the use of dynamically looking up the DNS suffix has been replaced with the static value of amazonaws.com. This does not appear to change by partition and instead requires users to set this manually for non-commercial regions.
• The default value for kms_key_enable_default_policy has changed from false to true to align with the default behavior of the aws_kms_key resource
• The Karpenter default value for create_instance_profile has changed from true to false to align with the changes in Karpenter v0.32
• The Karpenter variable create_instance_profile default value has changed from true to false. Starting with Karpenter v0.32.0, Karpenter accepts an IAM role and creates the EC2 instance profile used by the nodes

Removed

• The complete example has been removed due to its redundancy with the other examples
• References to the IRSA sub-module in the IAM repository have been removed. Once https://github.com/clowdhaus/terraform-aws-eks-pod-identity has been updated and moved into the organization, the documentation here will be updated to mention the new module.

Motivation and Context

• Resolves Windows Managed Node Group support #2350
• Resolves error configmap "aws-auth" does not exist and creating IAM Policy ekstest-additional: ConcurrentModification: Found Tagris has different internal #2525
• Resolves I/O Timeout when trying to create the aws-auth configmap #2541
• Resolves Deprecated warning with new hashicorp/aws v5.0.1 provider #2635
• Resolves Karpenter IRSA IAM policy to be like blueprints one #2733
• Resolves Karpenter roles to follow new policy guide from new release v0.32.x #2809
• Resolves AccessDeniedException kms:DescribeKey #2816
• Resolves Support for instance_maintenance_policy block in self_managed_node_group_defaults / self_managed_node_groups #2830
• Resolves fix: Update description for karpenter "create" var #2848
• Resolves Add EKS Pod Identity support to the module #2850
• Resolves Add Amazon CloudWatch log_group_class Support for Cluster Log Group #2866
• Closes fix: Typo spot_interupt #2863
• Resolves Add latest parameters to ASG instance_refresh block #2886
• Resolves Reference to an abandoned provider - gavinbunney/kubectl #2888
• Resolves fix: Remove gavinbunney/kubectl provider from Karpenter example #2889

Breaking Changes

• Absolutely

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

[jbronn]

Thank you @bryantbiggs for the upgrade work! With it, I successfully used the karpenter module here with the latest Karpenter 0.33.1 release. However, I needed to apply policy updates to be in sync with the new v1beta controller policy. In particular, I added region statements, added spot instance support, and fixed typos.

[bryantbiggs]

awesome, thanks @jbronn - theres a few parts we're waiting on for this release, and one of those is aws/karpenter-provider-aws#5195. I'd prefer to follow the policy set by the upstream project instead of crafting something custom here so we'll see what the Karpenter team comes back with

[jbronn]

@bryantbiggs I think we're on the same page, as my changes were an attempt to sync with latest 0.33.1 policy rather than some custom one I made myself. In particular, there are significant issues with the current policy here that prevent Karpenter working at all:

• Typos in AllowScopedResourceTagging and AllowScopedDeletion policy statements: they should be using ResourceTag instead of RequestTag in conditions.
• The AllowScopedEC2InstanceActionsWithTags has wrong conditions specified.
• The AllowInterruptionQueueActions and AllowPassingInstanceRole statements should be referring to the SQS queue and role ARNs instead of hard-coding strings that don't match the managed terraform resources.

[antonbabenko]

LGTM, small minor concerns about resource renaming and we are good to go!

[antonbabenko]

This PR is included in version 20.0.0 🎉

[morganrowse]

A module tag has been added to the cluster control plane

Hi @bryantbiggs any particular reason for this change? I don't see any way to modify the behavior as its a hard set merge. We like to enforce our own tagging standards and this breaks that. PRs welcome?

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.