Skip to content

v1.2.0

Latest

Choose a tag to compare

@heyvaldemar heyvaldemar released this 15 Oct 08:01
· 14 commits to main since this release

The EKS path, applied to a real AWS account for the first time. Run 38695231460 on 7040995 applied 101 resources, brought all 19 Applications to Synced and Healthy on EKS 1.37, passed every check in tests/eks/run.sh, destroyed everything, found nothing left and estimated its own cost at about $0.20. The seven runs before it failed on things only AWS could show, about $1.09 between them by their own estimates; each is a fix below with a check behind it. The run now repeats every Wednesday.

Fixed

  • The seventh validation run brought 18 of the platform's 19 Applications to Synced and Healthy on EKS, then its teardown left a network load balancer, two controller security groups and two volumes behind, and they kept the VPC from being deleted (cleaned by hand within the hour; the residue search then found nothing). eks-cleanup.sh had deleted the root Application, which cascaded through the AWS Load Balancer Controller before the Gateway's Service was gone. It now stops Argo CD first, deletes Gateways (Envoy Gateway would put their Services back), the remaining LoadBalancer Services, Ingresses and Karpenter's NodePools and NodeClaims while their controllers run. scripts/aws-controller-cleanup.sh then deletes from AWS's side, by the controllers' tags, what is still there: load balancers and target groups before the cluster goes, instances, security groups and available volumes after it, before the network. The run's teardown and make eks-down destroy in that order. diagnose.sh now shows certificates, ACME orders and challenges and Gateways.

  • What kept Argo CD's root Application Unknown on EKS 1.37: the default metrics-server add-on, v0.9.0-eksbuild.12, is built with Go 1.27, whose default jsonv2 experiment makes kube-openapi leak field names into the schema it publishes. The API server then answers /openapi/v2 as protobuf with 503 while JSON still works, and Argo CD reads it as protobuf (metrics-server#1868). The add-on is held at v0.9.0-eksbuild.11, the last build on Go 1.26 (go1.26.6, read from the binary; eksbuild.12 reads go1.27.2), with a test and a planted violation. bootstrap.sh now asks for the OpenAPI document the way Argo CD does and fails at once, naming the issue, if the answer is not 200; diagnose.sh prints the same answer.

  • On a new EKS cluster, Argo CD's controller failed its first cache sync with "error getting openapi resources: the server is currently unable to handle the request": it started while the cluster's aggregated metrics.k8s.io API was not answering yet, and left the root Application Unknown. bootstrap/bootstrap.sh now waits until every APIService is Available before it installs Argo CD and names any that is not, and diagnose.sh lists APIServices that are not Available. The fourth run also met a 429 "Data limit exceeded" pulling Karpenter's chart from public.ecr.aws anonymously; the validation run now logs in to ECR Public with its own role first.

  • scripts/aws-residue.sh failed a teardown that had left nothing: the Resource Groups Tagging API kept listing a deleted NAT gateway, subnet, security group and flow log, and three KMS keys scheduled for deletion. Each tagged ARN is now asked about directly, and only what still exists and is not on its way out counts; a type the script cannot ask about is still reported. The third run cost about $0.31 by the run's own estimate.

  • The third validation run reached a running EKS cluster with Argo CD installed and waited 45 minutes on a root Application whose sync stayed Unknown, then saved a table that did not say why. scripts/wait-for-platform.sh now stops after ten minutes when the root Application has not read the repository, and prints each pending Application's conditions, where Argo CD says why; the run's evidence now carries the Applications as YAML, the repo server's and application controller's logs and diagnose.sh.

  • The first real apply found two things the mocked-provider tests could not. Karpenter's EventBridge rule for capacity reservation interruptions was named past AWS's 64-character limit once the cluster name carried a run ID (74 characters); the key is shorter now, and a posture test checks every named resource against its AWS limit with the longest cluster name the variable allows, with a planted violation. And the guardrail that keeps workflow roles off the bootstrap's own IAM denied reads as well, so the EKS Terraform could not resolve its own role (aws_iam_session_context calls iam:GetRole); it now denies every IAM action on them but Get* and List*, with a test and a planted violation.

  • bootstrap-aws: the roles trusted the OIDC subject repo:owner/name:environment:..., and the first real validation run was refused at the first AWS call. GitHub gives repositories created or renamed after 15 July 2026 an immutable subject, repo:owner@OWNER_ID/name@REPO_ID:..., so a name someone gives up and someone else takes cannot match. The new subject_repository variable holds that form, defaults to this repository's IDs, accepts the legacy form for older repositories, and must name the same repository as repository. Two tests and a planted violation; nothing had been created in the account.

  • bootstrap-aws: the budget counted credits, as AWS Budgets does by default, so in an account running on promotional credits the $5 and $10 alerts and the $15 kill switch would never fire, and a run left running would spend the credits instead. It now measures usage before credits (cost_types.include_credit = false), with a test and a planted violation. The sweeper's daily table leaves credits and refunds out for the same reason.

Changed

  • The EKS validation run is weekly as well as by hand (ADR 9 and ADR 10, revisited): every Wednesday, after Monday's Dependabot updates have merged, and only while the repository variable AWS_VALIDATION_ENABLED is true, so it stops without a commit. The bootstrap's alerts move to $10 and $20 and its kill-switch to $30, the ceiling for a month of weekly runs.
  • ADR 10 and the bootstrap README allow a member account in the maintainer's own AWS Organization, and say why an account that holds other work is the wrong place: the workflow roles can delete what they did not make.
  • The README points at kubernetes-postgres-recovery, the database a tenant would run here, with the point-in-time recovery that Velero's volume snapshots cannot give it.

Added

  • Live status: every e2e run uploads its checks and measurements (E2E_RESULTS), and .github/workflows/status.yml rewrites the status branch after each run with the last 30 days on main: green days, the streak, the latest checks (scripts/status.py). The README's second row of badges reads it.

How this release was checked

On AWS: run 38695231460 on 7040995, every EKS check passing and nothing left after teardown. On kind: the fresh install and the upgrade from v1.1.0, every check, and Verification, on the commit this release is tagged on.

Full history in CHANGELOG.md.