test(e2e): credential-free mock E2E suite + real-AWS CI cap (1 PR / 2 post-merge / weekly matrix)#844
Merged
Conversation
…terfaces AWS ships no official mocks for aws-sdk-go-v2; its documented testing guidance is consumer-defined interfaces + DI + own fakes. This fake implements holodeck's internal/aws interfaces (39 EC2 + 12 ELBv2 + 1 SSM methods) over a mutex-guarded in-memory store with realistic lifecycle semantics: SDK-waiter-compatible instance states, exact *.NotFound error codes the provider's retry branches string-match, create/describe/delete consistency, error injection (FailNext) and a call/input recorder. Compile-time interface assertions make SDK shape changes fail the build. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…ten mocks Replaces the 494-line per-method-closure MockEC2Client (and the local MockELBv2Client in nlb_delete_test.go) with the shared stateful fake. Coverage envelope preserved 1:1 (155 subtests + 84 ginkgo specs before and after); one previously vacuous owner-id assertion made real; adds mutation-verified pinning tests for SG cross-reference revocation and NLB listener->target-group->LB teardown order. Shared helpers move to helpers_test.go. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…n seam 11 specs (label 'mock') drive the real Provider.Create/Delete/DryRun code against awsfake in ~2.4s with no AWS credentials or SSH key: lifecycle for 4 topologies (single-node, minimal, dedicated-CP, HA+NLB), rollback-on-injected-failure at 5 create phases asserting zero orphaned resources, cluster partial-failure cleanup via Delete, idempotent delete. newProvider gains an aws.Option passthrough; the suite-wide E2E_SSH_KEY requirement moves to a per-spec requireRealAWSEnv guard; the ginkgo CLI is pinned to the go.mod version instead of @latest. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
All 24 real-AWS entries gain 'real-aws' as their first label (existing labels preserved verbatim) and both DescribeTable bodies fail fast via requireRealAWSEnv when E2E_SSH_KEY is absent — keyless runs abort in seconds with zero AWS API activity instead of failing mid-provision. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…e cluster, add weekly matrix
PR pushes: mock suite (secret-free) + exactly 1 real job
('real-aws && default && !rpm', was 3 real jobs). Merges to
main/release: mock suite + 1 real minimal-cluster job + the existing
issue-on-failure automation. The displaced 15-label real matrix
(including nvidia-driver, previously smoke's third label) plus arm64
and integration-test move to e2e-weekly.yaml: weekly cron + manual
workflow_dispatch with an env-indirected label_filter input.
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…tiers Documents the three CI tiers, the credential-free mock suite and its local run command, label taxonomy (real-aws + topic labels; post-merge is currently informational only), and how to add a test so it actually runs in a scheduled tier. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Coverage Report for CI Build 29030211153Coverage increased (+1.9%) to 50.689%Details
Uncovered Changes
Coverage Regressions10 previously-covered lines in 2 files lost coverage.
Coverage Stats
💛 - Coveralls |
…ace nested lists mdl (MD055/MD057) does not parse tables indented under list items, and the repo style pins MD007 unordered-list indent to 2 spaces. Verified against the CI linter exactly: mdl 0.13.0 with docs/mdl-style.rb, rc=0. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
This was referenced Jul 9, 2026
Closed
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every CI run provisions real AWS infrastructure: 3 real E2E jobs per PR push and a
14-job real matrix (plus arm64 + integration-test) per merge — ~60 min wall time and
real EC2/EIP/GPU-instance cost per pipeline, with credentials required even for fork
PRs. Meanwhile, failure modes that real E2E can't afford to exercise (API errors
mid-create, rollback, idempotent teardown) had no coverage at all.
Approach
AWS ships no official mocks for aws-sdk-go-v2; its documented testing guidance is
consumer-defined interfaces + DI + your own fakes — exactly the shape holodeck
already has (
internal/aws.{EC2Client,SSMClient,ELBv2Client}+WithEC2Client/...options). This PR:
internal/aws/awsfake— one stateful, mutex-guarded in-memory fakeimplementing all three interfaces (typed against the official SDK structs;
compile-time assertions make SDK upgrades fail loudly). Realistic lifecycle
semantics: ID generation, instance state transitions compatible with the SDK
waiters, exact AWS
*.NotFounderror codes that the delete/retry paths string-match,error injection (
FailNext) and a call/input recorder.tests/e2e_mock_test.go, labelmock) — 11 specs driving thereal
Provider.Create/Delete/DryRuncode through the fake, credential-free, ~2.4s:full lifecycle for 4 topologies (single-node, minimal, dedicated-CP, HA+NLB),
rollback-on-injected-failure at 5 create phases (asserts zero orphaned resources),
cluster partial-failure cleanup, idempotent delete.
real-aws;E2E_SSH_KEYis now asserted per-real-spec (fail-fast) instead ofsuite-wide, so mock runs need nothing.
Merge to main/release: mock suite + 1 real minimal cluster + the existing
issue-on-failure job. New
e2e-weekly.yaml: the displaced 15-label real matrix(incl. nvidia-driver) + arm64 + integration-test, weekly cron + manual
workflow_dispatchwith a label-filter input.MockEC2Client(deleted) onto the same fake — coverage envelope preserved 1:1, one previously
vacuous assertion made real, plus new pinning tests for SG-revoke and NLB
teardown-order behavior. Harness modernization: ginkgo CLI pinned to the go.mod
version, JSON reports, no sleeps.
Testing done
go test ./pkg/provider/aws/... -count=1: 157 Go tests + 84 Ginkgo specs, 0 failuresgo test ./internal/aws/awsfake/ -count=1: 34 tests (fake behavior contract)env -u AWS_* -u E2E_SSH_KEY): 11/11 in ~2.4s--label-filter='real-aws' --dry-run: 24 specs selected; smoke and post-mergefilters each select exactly 1 spec
zero AWS API activity (guard proven)
behavior each turned the corresponding assertions red (rollback, NotFound codes,
ENI drain, owner-id filter, teardown order, fault-injection seam)
Cost/latency impact
Breaking changes
None to the CLI or provider API. CI behavior: rpm/dra/kernel/git-source real coverage
moves from per-merge to weekly + on-demand dispatch.
Known follow-ups (intentionally out of scope)
arm64weekly job is a pre-existing zero-spec no-op (no spec carries that label) — kept verbatim; needs a real arm64 spec or removal.post-mergelabel is now informational only (documented as such).create-issuejobs share thee2e-failurededup label — weekly failures comment onto open post-merge issues.workflow_dispatchruns nothing silently; dispatch failures don't file issues.MockEC2Clientremains inpkg/testutil/mocks(used bypkg/cleanup) — this PR scopes topkg/provider/aws.createPrivateRouteTable/createNATGateway, zero callers) — candidate for removal.Reviewer FAQ
Why not LocalStack/moto? Community LocalStack EC2 is emulation-only — no SSH-able
VMs, so holodeck's provisioner phase can't run against it either; it adds a Docker
dependency and ~30s startup per job for no coverage the fake doesn't give. The fake is
typed against the official SDK structs and follows AWS's documented Go SDK v2 testing
pattern.
How do we catch fake-vs-real drift? SDK shape changes: compile-time interface
assertions. Semantic drift: the 2 real jobs per merge + weekly matrix; policy — a
mock-green/real-red divergence is treated as a fake bug, not a flake.
Who owns weekly failures? The weekly create-issue job files/updates a GitHub
issue; suggest assigning a triage owner so the matrix doesn't rot (risk register).
Merge strategy note for maintainers: the branch is 6 logical commits, each
building and testing green standalone — squash and rebase are both safe.