Skip to content

Releases: heyvaldemar/aws-eks-reference-platform

v1.2.0

Choose a tag to compare

@heyvaldemar heyvaldemar released this 15 Oct 08:01

The EKS path, applied to a real AWS account for the first time. Run 38695231460 on 7040995 applied 101 resources, brought all 19 Applications to Synced and Healthy on EKS 1.37, passed every check in tests/eks/run.sh, destroyed everything, found nothing left and estimated its own cost at about $0.20. The seven runs before it failed on things only AWS could show, about $1.09 between them by their own estimates; each is a fix below with a check behind it. The run now repeats every Wednesday.

Fixed

  • The seventh validation run brought 18 of the platform's 19 Applications to Synced and Healthy on EKS, then its teardown left a network load balancer, two controller security groups and two volumes behind, and they kept the VPC from being deleted (cleaned by hand within the hour; the residue search then found nothing). eks-cleanup.sh had deleted the root Application, which cascaded through the AWS Load Balancer Controller before the Gateway's Service was gone. It now stops Argo CD first, deletes Gateways (Envoy Gateway would put their Services back), the remaining LoadBalancer Services, Ingresses and Karpenter's NodePools and NodeClaims while their controllers run. scripts/aws-controller-cleanup.sh then deletes from AWS's side, by the controllers' tags, what is still there: load balancers and target groups before the cluster goes, instances, security groups and available volumes after it, before the network. The run's teardown and make eks-down destroy in that order. diagnose.sh now shows certificates, ACME orders and challenges and Gateways.

  • What kept Argo CD's root Application Unknown on EKS 1.37: the default metrics-server add-on, v0.9.0-eksbuild.12, is built with Go 1.27, whose default jsonv2 experiment makes kube-openapi leak field names into the schema it publishes. The API server then answers /openapi/v2 as protobuf with 503 while JSON still works, and Argo CD reads it as protobuf (metrics-server#1868). The add-on is held at v0.9.0-eksbuild.11, the last build on Go 1.26 (go1.26.6, read from the binary; eksbuild.12 reads go1.27.2), with a test and a planted violation. bootstrap.sh now asks for the OpenAPI document the way Argo CD does and fails at once, naming the issue, if the answer is not 200; diagnose.sh prints the same answer.

  • On a new EKS cluster, Argo CD's controller failed its first cache sync with "error getting openapi resources: the server is currently unable to handle the request": it started while the cluster's aggregated metrics.k8s.io API was not answering yet, and left the root Application Unknown. bootstrap/bootstrap.sh now waits until every APIService is Available before it installs Argo CD and names any that is not, and diagnose.sh lists APIServices that are not Available. The fourth run also met a 429 "Data limit exceeded" pulling Karpenter's chart from public.ecr.aws anonymously; the validation run now logs in to ECR Public with its own role first.

  • scripts/aws-residue.sh failed a teardown that had left nothing: the Resource Groups Tagging API kept listing a deleted NAT gateway, subnet, security group and flow log, and three KMS keys scheduled for deletion. Each tagged ARN is now asked about directly, and only what still exists and is not on its way out counts; a type the script cannot ask about is still reported. The third run cost about $0.31 by the run's own estimate.

  • The third validation run reached a running EKS cluster with Argo CD installed and waited 45 minutes on a root Application whose sync stayed Unknown, then saved a table that did not say why. scripts/wait-for-platform.sh now stops after ten minutes when the root Application has not read the repository, and prints each pending Application's conditions, where Argo CD says why; the run's evidence now carries the Applications as YAML, the repo server's and application controller's logs and diagnose.sh.

  • The first real apply found two things the mocked-provider tests could not. Karpenter's EventBridge rule for capacity reservation interruptions was named past AWS's 64-character limit once the cluster name carried a run ID (74 characters); the key is shorter now, and a posture test checks every named resource against its AWS limit with the longest cluster name the variable allows, with a planted violation. And the guardrail that keeps workflow roles off the bootstrap's own IAM denied reads as well, so the EKS Terraform could not resolve its own role (aws_iam_session_context calls iam:GetRole); it now denies every IAM action on them but Get* and List*, with a test and a planted violation.

  • bootstrap-aws: the roles trusted the OIDC subject repo:owner/name:environment:..., and the first real validation run was refused at the first AWS call. GitHub gives repositories created or renamed after 15 July 2026 an immutable subject, repo:owner@OWNER_ID/name@REPO_ID:..., so a name someone gives up and someone else takes cannot match. The new subject_repository variable holds that form, defaults to this repository's IDs, accepts the legacy form for older repositories, and must name the same repository as repository. Two tests and a planted violation; nothing had been created in the account.

  • bootstrap-aws: the budget counted credits, as AWS Budgets does by default, so in an account running on promotional credits the $5 and $10 alerts and the $15 kill switch would never fire, and a run left running would spend the credits instead. It now measures usage before credits (cost_types.include_credit = false), with a test and a planted violation. The sweeper's daily table leaves credits and refunds out for the same reason.

Changed

  • The EKS validation run is weekly as well as by hand (ADR 9 and ADR 10, revisited): every Wednesday, after Monday's Dependabot updates have merged, and only while the repository variable AWS_VALIDATION_ENABLED is true, so it stops without a commit. The bootstrap's alerts move to $10 and $20 and its kill-switch to $30, the ceiling for a month of weekly runs.
  • ADR 10 and the bootstrap README allow a member account in the maintainer's own AWS Organization, and say why an account that holds other work is the wrong place: the workflow roles can delete what they did not make.
  • The README points at kubernetes-postgres-recovery, the database a tenant would run here, with the point-in-time recovery that Velero's volume snapshots cannot give it.

Added

  • Live status: every e2e run uploads its checks and measurements (E2E_RESULTS), and .github/workflows/status.yml rewrites the status branch after each run with the last 30 days on main: green days, the streak, the latest checks (scripts/status.py). The README's second row of badges reads it.

How this release was checked

On AWS: run 38695231460 on 7040995, every EKS check passing and nothing left after teardown. On kind: the fresh install and the upgrade from v1.1.0, every check, and Verification, on the commit this release is tagged on.

Full history in CHANGELOG.md.

v1.1.0

Choose a tag to compare

@heyvaldemar heyvaldemar released this 14 Oct 01:52

Onboarding as a command, and the AWS account the EKS validation run needs, written and tested but not yet applied. Nothing here has run against a real AWS account; the README's EKS section says so and will carry the date and commit of the first run that passes.

Added

  • kubectl tenant (tools/kubectl-tenant). new <team> writes apps/<team> from the template and refuses the names the tenant chart refuses, read from platform/tenant/values.yaml so the two lists cannot drift. check <team> reports from the files alone what admission would refuse: an image without a digest, missing requests, no memory limit, missing probes, a privileged container, the host network. Its tests rebuild apps/team-b, which was made by hand, and fail if the command's result differs.
  • terraform/bootstrap-aws, the account ADR 10 asks for. GitHub OIDC roles trusted per environment; a creation role held by guardrails and by a permissions boundary every role it creates must carry; a sweep role that reads, deletes and rewrites a run's state but creates nothing; budget alerts at $5 and $10, and a kill switch at $15 that attaches deny-all to the creation role. Each guardrail is shown a planted violation it must catch.
  • The EKS validation run, dispatched by hand with confirm: SPEND. Seven checks against the live cluster, teardown in a job that runs whatever the first one did, a residue check that fails on anything the run left, and a cost report from resource-hours. A daily sweeper deletes what a broken run left; it stays off until the account exists.
  • Terraform: zone_id for an existing, delegated Route 53 zone; iam_permissions_boundary on all seven IAM roles the module creates.
  • docs/engagement/: executive summary, discovery, a success plan whose eleven criteria map to e2e steps, an adoption plan from Docker Compose to a tenant, sizing, handover.
  • CodeQL (security-extended) for Python, Go and the workflows.

Changed

  • Charts: argo-cd 10.10.3, kube-prometheus-stack 92.3.1, opentelemetry-collector 0.176.0. The rendered manifests differ only in the chart version labels.
  • scripts/diagnose.sh prints the conditions of the object a waiting Argo CD sync waits on. Argo CD 3 keeps no per-resource health in the Application, so a timed-out run used to say which object and never why.
  • ADR 5 cites kyverno/kyverno#6554; the README lists its known limitations in one place.

Security

  • bootstrap-aws, before it was ever released: the creation role could create a role without the guardrails, or rewrite the guardrails policy, and act through either. Every role it creates must now carry the permissions boundary, the bootstrap's own roles and policies are out of its reach, and the roles it may touch are limited to the run's prefix. Found in review; no account was created with the gap. What it still does not stop is in terraform/bootstrap-aws/README.md.

How this release was checked

On the commit before this one: kind e2e, 56 of 56 checks on a fresh install and 56 of 56 after upgrading a cluster from v1.0.1, and Verification, with every planted violation caught: 25 in the rendered manifests, 25 in the main Terraform module, 10 in bootstrap-aws.

Full history in CHANGELOG.md.

v1.0.1

Choose a tag to compare

@heyvaldemar heyvaldemar released this 13 Oct 12:57

A fix release, found by the first upgrade test to install a release by its tag.

Fixed

  • The repository token is stored without a trailing newline. bootstrap.sh fed PLATFORM_REPO_TOKEN to kubectl through a here-string, which appends one. Argo CD resolves a tag or a branch by listing the remote's refs, and that listing sent the stored token and was refused with "Invalid username or token". A commit SHA needs no listing, so the e2e run, pinned to a commit, passed; the upgrade job, installing v1.0.0 by its tag, failed twice. Affects a repository read over HTTPS with PLATFORM_REPO_TOKEN set, public or private, at a tag or a branch. Without a token, or with a deploy key, it was not affected.

Added

  • e2e step 8: Argo CD resolves the latest release tag with the credential the cluster holds, on a fresh install and after the upgrade.

Changed

How this release was checked

On the commit before this one: kind e2e, 56 of 56 checks on a fresh install and 56 of 56 after upgrading a cluster from v1.0.0, and Verification.

Full history in CHANGELOG.md.

v1.0.0

Choose a tag to compare

@heyvaldemar heyvaldemar released this 12 Oct 22:18

The first release of a reference Kubernetes platform for Amazon EKS. Everything it claims is a check that fails the build when the claim stops being true, and every check is shown the violation it exists for.

What is in it

  • The platform as Argo CD ApplicationSets keyed on the cluster Secret, every chart at one exact version and every Git source at the cluster's pinned commit: Envoy Gateway, cert-manager, External Secrets, Kyverno, kube-prometheus-stack, the OpenTelemetry Collector and Velero on every cluster; Karpenter, the AWS Load Balancer Controller and ExternalDNS on EKS.
  • Admission the API server enforces itself. Kyverno CEL policies compiled into native ValidatingAdmissionPolicies: images by digest, requests and memory limits, probes, restricted Pod Security for tenants. They keep refusing with Kyverno scaled to zero, which deletes its own webhooks on the way down.
  • Tenants, one directory each. The platform creates each tenant's namespace, quota and Argo CD project; the project has one destination and an allow-list of kinds with no RBAC on it, and every tenant sync runs as an account the API server limits to the same kinds (Argo CD sync impersonation, enforced).
  • Backups proven by restores. Velero file-system backup, a daily drill that deletes the demo tenant and its volume, restores it and reads back data written before the backup.
  • An SLO that is proven to page, with multi-window burn-rate alerts unit-tested by promtool and made to fire in the cluster.
  • Terraform for EKS 1.37: access entries only, KMS for Secrets, IMDSv2 with hop limit 1, Pod Identity per controller, Graviton system nodes, Karpenter's AWS side, the Velero bucket. Tested with terraform test against a mocked provider, 24 planted violations.
  • Thirteen decision records (docs/decisions/), each ending with how the choice would be explained to a customer; a threat model; a Well-Architected review; runbooks; and a hands-on guide whose three break-and-fix exercises run as written every week.

What it does not claim

The EKS path has not been applied to a real AWS account. It is written, validated, linted and tested against a mocked provider; the guarded, cost-reported real-cloud run is designed (ADR 10) and not built. Nothing here runs in production anywhere. The README's "What is proven, and how" table is the exact list of what is.

How this release was checked

On d439a41, the commit before this release's changelog commit:

  • kind e2e: a fresh kind cluster, the whole platform installed by Argo CD, every Application Synced and Healthy, then 55 of 55 checks. The first attempt timed out after 25 minutes waiting for kube-prometheus-stack to report Healthy, on a runner whose node had gone NotReady; the rerun passed. It is recorded here because a release note that hides a red run is the kind of status this platform exists to distrust.
  • Verification: the rendered manifests through kubeconform, kube-linter, pluto, the Argo CD and policy checks and promtool, 25 planted violations each caught, and the Terraform tests with their 24.
  • hands-on guide, on e6654d3: the three exercises as written. The only later change touches one request in the e2e suite.

Upgrading

This is the first release. From the next one on, CI installs the previous release, writes data, moves the cluster to the new commit and reads the data back before a release is cut.

Full history in CHANGELOG.md.