Skip to content

feat(tfacc): run upstream terraform-provider-aws acc tests against fakecloud#369

Merged
vieiralucas merged 4 commits intomainfrom
worktree-tfdrift-harness
Apr 14, 2026
Merged

feat(tfacc): run upstream terraform-provider-aws acc tests against fakecloud#369
vieiralucas merged 4 commits intomainfrom
worktree-tfdrift-harness

Conversation

@vieiralucas
Copy link
Copy Markdown
Member

@vieiralucas vieiralucas commented Apr 14, 2026

Summary

  • New fakecloud-tfacc crate runs hashicorp/terraform-provider-aws TestAcc* against a live fakecloud. This is the semantic test layer PR Fix DynamoDB DescribeTable warm throughput response #362 highlighted as a gap: pure SDK round-trip tests tolerate missing optional fields, but the Terraform provider enforces waiters, field presence, and drift. Now we catch that class of bug in CI.
  • Two-layer opt-in: a service allow-list (only services we claim to support get run) plus a per-service deny-list for upstream tests that are structurally unsupportable or track a known fakecloud gap. Each deny entry has a reason comment grouped by category (`unsupportable`, `gap`, `hung`).
  • Batch 1 ships DynamoDB coverage: 38 TestAccDynamoDBTable_* tests green against fakecloud in ~2.5 min locally; 38 tests denied with reasons (Global Tables, INFREQUENT_ACCESS, GSI update gaps, deletion_protection, etc). Future batches widen the allow-list and drive the `gap:` entries to zero.
  • CI workflow fans out one runner per allow-listed service (matrix generated by a tiny `tfacc_services` helper binary, same pattern as `e2e.yml`). Each runner installs Go 1.23 + Terraform 1.14, builds fakecloud, and runs `go test -parallel 8` inside one `cargo test` invocation. Caches cover `/.cache/go-build`, `/go/pkg/mod`, and the shallow provider checkout in `target/tfacc`.
  • Fix surfaced during triage: DynamoDB `DescribeTable` no longer emits an `SSEDescription` block for tables using the default AWS-owned key. Real AWS omits the field in that case; the previous stub made even `TestAccDynamoDBTable_basic` fail on the provider's drift check.

How it works

  1. First test run shallow-clones `terraform-provider-aws@v5.97.0` into `target/tfacc/` (idempotent) and strips one `godebug tlskyber=0` directive that Go ≥1.24 rejects.
  2. Each `#[tokio::test]` spawns fakecloud on a random port and invokes `go test ./internal/service// -run -skip ` with `TF_ACC=1`, dummy credentials, and `AWS_ENDPOINT_URL_*` env vars pointing at the local endpoint.
  3. Failure → `assert_pass` panics with the tail of `go test` output so CI shows which upstream test broke and why. Success → silent pass.
  4. Missing `go` or `terraform` → tests skip cleanly (same pattern as `FAKECLOUD_CONTAINER_CLI=false` in `fakecloud-e2e`).

Test plan

  • `cargo test -p fakecloud-tfacc dynamodb_acceptance` green locally in 153s (38 upstream tests via go test)
  • `cargo clippy -p fakecloud-tfacc --all-targets -- -D warnings` clean
  • `cargo test -p fakecloud-dynamodb` still green (93 unit tests) after the SSEDescription fix
  • `cargo check --workspace` clean
  • New `tfacc` CI workflow green on this PR
  • Cubic review clean

Summary by cubic

Adds fakecloud-tfacc to run upstream hashicorp/terraform-provider-aws acceptance tests against a live fakecloud. First batch enables DynamoDB with 38 passing table tests and fixes SSE parity to prevent Terraform drift.

  • New Features

    • New fakecloud-tfacc crate runs upstream go test against fakecloud via AWS_ENDPOINT_URL_*.
    • Two-layer gating: service allow-list plus per-service deny-list with reasons; DynamoDB batch 1 runs TestAccDynamoDBTable_* (38 pass, 38 skipped).
    • Shallow-clones terraform-provider-aws@v5.97.0 into target/tfacc/ and strips the Go 1.24-incompatible godebug directive.
    • New CI workflow fans out per service (matrix from tfacc_services), caches ~/.cache/go-build, ~/go/pkg/mod, and target/tfacc, and excludes fakecloud-tfacc from generic ci.yml/release.yml test jobs so it runs only in tfacc.yml.
    • Hard-fails with a clear message when go or terraform are missing.
  • Bug Fixes

    • DynamoDB: DescribeTable no longer returns SSEDescription when using the default AWS-owned key; updated e2e test to assert omission to match AWS and fix provider drift checks.
    • TFACC: avoid unnecessary go.mod rewrites when no godebug tlskyber line exists; preserve trailing newlines during patching.

Written for commit 6afe4b0. Summary will update on new commits.

…kecloud

New `fakecloud-tfacc` crate invokes `hashicorp/terraform-provider-aws`
acceptance tests (`TestAcc*`) against a live fakecloud instance. This is
the semantic test layer that catches waiter / field-presence / drift
bugs like PR #362, which pure SDK round-trip tests miss.

- Crate shallow-clones terraform-provider-aws@v5.97.0 into `target/tfacc/`
  on first run and patches `go.mod` to remove a go1.24-incompatible
  godebug directive. Skips cleanly when go or terraform are missing.
- Two-layer opt-in: `SERVICES` allow-list for services; each service
  carries a deny-list for upstream tests that are either structurally
  unsupportable or track a known fakecloud gap. Every deny entry has a
  reason comment.
- One `#[tokio::test]` per allow-listed service, invoking
  `go test -parallel 8` with `AWS_ENDPOINT_URL_*` env vars pointed at
  a fresh fakecloud process per Rust test.
- CI workflow fans out one runner per service (matrix generated by the
  new `tfacc_services` helper binary, same pattern as e2e.yml) and
  caches `~/.cache/go-build`, `~/go/pkg/mod`, and `target/tfacc` so the
  provider checkout survives across runs.
- Batch 1 covers DynamoDB: 38 `TestAccDynamoDBTable_*` tests green
  against fakecloud in ~2.5 min locally; 38 tests denied with reasons
  (Global Tables, INFREQUENT_ACCESS, streams gaps, GSI update gaps,
  deletion_protection, backup encryption, etc). Later batches expand
  the allow-list and drive the "gap" deny entries to zero.
- Small fakecloud fix revealed during triage: DynamoDB DescribeTable
  no longer emits an `SSEDescription` block for tables using the
  default AWS-owned key. Real AWS omits the field in that case; the
  previous stub made `TestAccDynamoDBTable_basic` fail on the drift
  check.

Follow-ups tracked in `crates/fakecloud-tfacc/src/allowlist.rs`:
- widen the dynamodb `run_regex` beyond `TestAccDynamoDBTable_`
- drive the "gap:" deny entries for dynamodb to zero
- add sqs and other services to the allow-list
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 10 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/fakecloud-tfacc/src/lib.rs">

<violation number="1" location="crates/fakecloud-tfacc/src/lib.rs:99">
P2: `strip_godebug` rewrites `go.mod` on every call because the comparison ignores trailing newline behavior from `lines()`.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread crates/fakecloud-tfacc/src/lib.rs Outdated
- strip_godebug: skip the write when no godebug line exists (Cubic P2
  finding). The previous comparison swallowed the file's trailing
  newline via `lines()`, causing the harness to rewrite go.mod on
  every invocation even when nothing changed.
- e2e: rename `dynamodb_sse_default_aes256` to
  `dynamodb_sse_default_omitted` and assert the new, correct behaviour
  — DescribeTable no longer emits an SSEDescription block when the
  table uses the default AWS-owned key.
…skip

Silently passing when the toolchain is absent just hides regressions —
if someone runs this crate they've opted into exercising the upstream
Terraform suite, and a missing binary should be a loud failure with
install instructions, not a green check on a machine that never ran
the code.

Replaces `toolchain_available() -> bool` with `require_toolchain()`
which panics with an actionable message naming the missing binaries.
The new tfacc crate hard-fails when go/terraform are missing (by
design — silently skipping would hide regressions). The generic
`cargo test --workspace` job in ci.yml/release.yml doesn't install
that toolchain, so it must opt out the same way it already opts out
of fakecloud-e2e and fakecloud-conformance.
@vieiralucas vieiralucas merged commit 2abeccd into main Apr 14, 2026
23 checks passed
@vieiralucas vieiralucas deleted the worktree-tfdrift-harness branch April 14, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant