MEDIUM: AWS Terraform + `eksctl`

[ranchodeluxe]

Background:

We have decisions to make here:

• It's pretty standard to set up AWS EKS clusters these days through a mixture of TF and eksctl. eksctl handles both k8s and AWS IAM roles/policies in one fell swoop for many workflows. That means it's not straight IAC but more of a documented walk through. We can take inspiration for this pattern from JupyterHub 2i2c docs

• Or... we can choose to give users a smoother IAC flow and do it all in TF but we'll have to pay particular attention to how nodes and other resources are tagged so they work seemlessly with all the EKS addons

Extras we'll want to include in the cluster:

• cluster OIDC provider (not for auth into the cluster but for IAM role delegation from k8s ServiceAccount roles)
• ebs csi driver addon
• aws-load-balancer-controller
• nginx controller

AC

• choose an option above to go with 1) eksctl + TF or 2) just TF
• write the code + docs

[geohacker]

@ranchodeluxe there are some previous work we could borrow from if you'd like though I'm not sure the state of that, and please feel free to ignore if it's not useful:

• recently @ingalls put together an EKS deployment just using cloudformation https://github.com/developmentseed/labs-eks/tree/main — no terraform here
• Alex Bush (ex devseed k8s advocate) started https://github.com/developmentseed/k8seed
• I know @batpad and @sunu are talking about this too

[ranchodeluxe]

@ranchodeluxe there are some previous work we could borrow from if you'd like though I'm not sure the state of that, and please feel free to ignore if it's not useful:

* recently @ingalls put together an EKS deployment just using cloudformation https://github.com/developmentseed/labs-eks/tree/main — no terraform here

* Alex Bush (ex devseed k8s advocate) started https://github.com/developmentseed/k8seed

* I know @batpad and @sunu are talking about this too

• I think we'll want to stay away from CloudFormation unless we decide to use eksctl (which leverages it)
• k8seed is pretty much empty and is focused on Azure

[sunu]

@ranchodeluxe How do you feel about decoupling the cluster creation part out of this repo into its separate repo? I was talking to @batpad last week about taking over the existing k8s-cluster repo, adding terraform code and documentation about creating a GKE cluster and adding some guidelines for users on how to run their apps on the cluster (https://github.com/developmentseed/labs/issues/307#issuecomment-1586744471)