feat: Add support to prepare CoreDNS for Helm management, enable cluster proportional autoscaler for CoreDNS #1033

bryantbiggs · 2022-10-07T00:02:42Z

What does this PR do?

Add support to prep CoreDNS for Helm management within the CoreDNS addon. This improves the usability experience when deploying self-managed CoreDNS via the addon
Add support for CoreDNS cluster proportional autoscaler. This is now enabled by default with some sane default settings when using either EKS managed CoreDNS or self-managed via the addon
Add time_sleep resource to CoreDNS addon to ensure proper sequencing of events within the addon (all of the following are subject to being enabled, but regardless the order still stands now):
1. Remove the default deployment of CoreDNS
2. Deploy the self-managed (Helm addon) CoreDNS deployment
3. Enable cluster proportional autoscaler
Add new variable data_plane_wait_arn which can be used to coordinate the dependency ordering between the EKS cluster creation and addon creation. By passing in an ARN from a node group or Fargate profile (or an array of ARNs using join(), the time_sleep resource will wait for the ARN(s) passed before starting a brief sleep duration and then allowing the deployment of the addons. This makes it possible to do a single terraform apply with the proper ordering and at a later date we can pass additional information through time_sleep to ensure proper ordering when performing cluster upgrades without the need for additional information from users (*I believe - TBD).
Add support for the 2048 sample/demo app as an "addon" to be able to re-use in other examples. Removing the use of the kubectl provider for this and only using the Kubernetes provider reduces the number of providers needed in the examples for this demo app as well
Update Fargate example blueprint
- Enable 2048 demo app via addon
- Update CoreDNS to be self-managed to run on Fargate
- Show example of Fargate wildcard profile usage for the demo app
- Enable fargate-fluentbit addon for Fargate logging (removes warnings from pods + we should have logging)

Motivation

Improve user experience with CoreDNS as well as Fargate based usage
Resolves Opinionated defaults for CoreDNS #1028
Resolves Fargate wildcard profile support #891

More

Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
Yes, I have added a new example under examples to support my PR
Yes, I have created another PR for add-ons under add-ons repo (if applicable)
Yes, I have updated the docs for this feature
Yes, I ran pre-commit run -a with this PR

…DNS addon

…rgate example

bryantbiggs · 2022-10-07T00:03:59Z

examples/fargate-serverless/main.tf

      fargate_profile_namespaces = [
        {
-          namespace = local.sample_app_namespace
+          namespace = "app-*"


Resolves #891

bryantbiggs · 2022-10-07T00:04:31Z

examples/fargate-serverless/main.tf

@@ -103,6 +94,9 @@ module "eks_blueprints_kubernetes_addons" {
  eks_oidc_provider    = module.eks_blueprints.oidc_provider
  eks_cluster_version  = module.eks_blueprints.eks_cluster_version

+  # Wait on the `kube-system` profile before provisioning addons
+  data_plane_wait_arn = module.eks_blueprints.fargate_profiles["kube_system"].eks_fargate_profile_arn


pretty nifty - addons don't start provisioning until the profile has provisioned

do you find this approach better than the use of depends_on ?
I assume it replaces it, corrrect?

yes - because

You can specify for create or destroy operations separately

You can specify individual triggers which will cause the resource to re-evaluate https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep#triggers-usage

By using the triggers, you use their outputs downstream to better control what is updated whereas with depends_on its applied to all downstream computed values when used with modules Using depends_on in a module definition fails terraform plan when using data source aws_partition in the module hashicorp/terraform#30340

I can variablize what the dependent arguments are - whereas, with depends_on I cannot

In short, it has much better control than what depends_on offers, especially from a module usage perspective.

bryantbiggs · 2022-10-07T00:05:41Z

examples/fargate-serverless/main.tf

-# Modifying CoreDNS for Fargate
-#---------------------------------------------------------------
-
-locals {


All this lives in the CoreDNS addon now so users do not need to re-create if they want to use this method of assuming control of CoreDNS with Helm

bryantbiggs · 2022-10-07T00:06:15Z

examples/fargate-serverless/main.tf

@@ -252,31 +177,12 @@ module "vpc" {
  default_security_group_tags   = { Name = "${local.name}-default" }

  public_subnet_tags = {
-    "kubernetes.io/cluster/${local.name}" = "shared"


no longer required with recent versions of ALB controller

interesting. where is this documented?
we are on the latest release of AWS LB Controller, and still tag out SGs

https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/deploy/subnet_discovery.md#common-tag

Security group tags is separate, and that is still required. This is just subnet tags

right right right!

bryantbiggs · 2022-10-07T00:07:09Z

modules/kubernetes-addons/cluster-proportional-autoscaler/variables.tf

@@ -22,7 +22,5 @@ variable "addon_context" {
    eks_oidc_issuer_url            = string
    eks_oidc_provider_arn          = string
    tags                           = map(string)
-    irsa_iam_role_path             = string


askulkarni2

One question, other wise the PR is rock solid.

askulkarni2 · 2022-10-07T01:40:17Z

modules/kubernetes-addons/aws-coredns/main.tf

+      # This simply maps a dependency for Terraform to ensure the default deployment is
+      # removed first (if enabled) before provisioning self-managed version (if enabled)
+      name  = "blueprints.connection"
+      value = try(null_resource.remove_default_coredns_deployment[0].id, "none")


askulkarni2 · 2022-10-07T01:45:34Z

modules/kubernetes-addons/variables.tf

@@ -15,6 +15,12 @@ variable "eks_worker_security_group_id" {
  default     = ""
 }

+variable "data_plane_wait_arn" {
+  description = "Addon deployment will not proceed until this value is known. Set to node group/Fargate profile ARN to wait for data plane to be ready before provisioning addons"


Should this be a list? What if I have both node groups and fargate profiles?

A list can still be used if you wrap it in a join() function. Ideally, you wouldn't want to wait for all of the node groups and Fargate profiles to come up, you just need one where addons can start deploying. Also, using one over using many will keep the potential for updates to a minimum (the more that are provided in the trigger, the more times there could be a re-trigger when its not necessary)

… into feat/coredns-opinions

askulkarni2

LGTM!

…ter proportional autoscaler for CoreDNS (aws-ia#1033) Co-authored-by: Apoorva Kulkarni <kuapoorv@amazon.com>

bryantbiggs added 3 commits October 5, 2022 19:25

feat: Add support to prep CoreDNS for Helm management within the Core…

8a6b18c

…DNS addon

feat: Add cluster-proportional-autoscaler to CoreDNS addon, update Fa…

51bb4e0

…rgate example

docs: Update docs for changes to CoreDNS, remove file not used

b92e71c

bryantbiggs requested review from askulkarni2, kcoleman731, vara-bonthu and Zvikan as code owners October 7, 2022 00:02

bryantbiggs had a problem deploying to EKS Blueprints Test October 7, 2022 00:02 Error

bryantbiggs commented Oct 7, 2022

View reviewed changes

fix: Update vault addon link

7abef5b

bryantbiggs temporarily deployed to EKS Blueprints Test October 7, 2022 00:10 Inactive

askulkarni2 reviewed Oct 7, 2022

View reviewed changes

bryantbiggs added 2 commits October 10, 2022 11:30

Merge branch 'main' of github.com:aws-ia/terraform-aws-eks-blueprints…

534402b

… into feat/coredns-opinions

fix: Updates from parallel e2e test run

3a1212a

bryantbiggs had a problem deploying to EKS Blueprints Test October 10, 2022 16:00 Error

Merge branch 'main' into feat/coredns-opinions

2410645

bryantbiggs temporarily deployed to EKS Blueprints Test October 10, 2022 16:55 Inactive

Merge branch 'main' into feat/coredns-opinions

f1cd420

askulkarni2 temporarily deployed to EKS Blueprints Test October 11, 2022 04:58 Inactive

askulkarni2 approved these changes Oct 11, 2022

View reviewed changes

askulkarni2 merged commit 37be09b into main Oct 11, 2022

askulkarni2 deleted the feat/coredns-opinions branch October 11, 2022 05:30

linusguan mentioned this pull request Dec 9, 2022

fix: Disable proportional autoscaler for CoreDNS for fully private cluster #1243

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add support to prepare CoreDNS for Helm management, enable cluster proportional autoscaler for CoreDNS #1033

feat: Add support to prepare CoreDNS for Helm management, enable cluster proportional autoscaler for CoreDNS #1033

bryantbiggs commented Oct 7, 2022 •

edited

bryantbiggs Oct 7, 2022

bryantbiggs Oct 7, 2022

FernandoMiguel Oct 7, 2022

bryantbiggs Oct 7, 2022 •

edited

bryantbiggs Oct 7, 2022

bryantbiggs Oct 7, 2022

FernandoMiguel Oct 7, 2022

bryantbiggs Oct 7, 2022

bryantbiggs Oct 7, 2022

FernandoMiguel Oct 10, 2022

bryantbiggs Oct 7, 2022

askulkarni2 left a comment

askulkarni2 Oct 7, 2022

askulkarni2 Oct 7, 2022

bryantbiggs Oct 7, 2022

askulkarni2 left a comment

feat: Add support to prepare CoreDNS for Helm management, enable cluster proportional autoscaler for CoreDNS #1033

feat: Add support to prepare CoreDNS for Helm management, enable cluster proportional autoscaler for CoreDNS #1033

Conversation

bryantbiggs commented Oct 7, 2022 • edited

What does this PR do?

Motivation

More

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantbiggs Oct 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

askulkarni2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

askulkarni2 left a comment

Choose a reason for hiding this comment

bryantbiggs commented Oct 7, 2022 •

edited

bryantbiggs Oct 7, 2022 •

edited