fix(cd): adopt an existing CD state bucket instead of creating a new one

[defangdevs]

Problem

The CodeBuild CD driver (src/pkg/clouds/aws/codebuild/cfn/) always creates a stack-managed, CFN-auto-named S3 bucket for Pulumi state, with RetainExceptOnCreate. Because the bucket is retained on stack delete but named per-stack-instance:

• defang cd teardown deletes the defang-cd stack → the bucket is retained (orphaned);
• the next connect/bootstrap creates a fresh stack → CFN mints a new defang-cd-bucket-* and the old one (plus its Pulumi state) is stranded.

Repeated over time this leaves many orphaned buckets in an account (observed: 11 defang-cd-bucket-* in one account, only 3 backed by a live stack). This is the same problem as #358, which #422 fixed for the old ECS CD — but that logic (findExistingBucket + a skip-bucket template override) did not survive the ECS→CodeBuild CD rewrite.

Fix

Before creating the stack, look for a retained CD bucket in the same account+region — buckets named <stack>-bucket-* carrying the defang:Prefix tag equal to the stack name. If exactly one exists, generate the template without a managed Bucket resource and emit that bucket as the bucketName output; otherwise create a managed bucket as before. >1 match errors rather than guessing.

RetainExceptOnCreate makes the one-time managed→adopted transition safe: when the Bucket resource leaves the template on the next update, CloudFormation retains the physical bucket, and the stack now simply references it.

• template.go: CreateTemplate(stack, existingBucket string) — skips the managed bucket and outputs the existing name when adopting.
• setup.go: findAdoptableBucket() (ListBuckets → prefix + region + defang:Prefix tag) wired into SetUp.
• The published/quick-create template (byoc/aws/byoc.go) is unchanged — it always creates a managed bucket (existingBucket == "").

Tests / verification

• New TestFindAdoptableBucket (mocked S3): none / one-match / wrong-region / wrong-tag / ambiguous.
• New TestCreateTemplateBucketAdoption: managed variant creates the bucket; adopt variant has no Bucket resource and outputs the existing name.
• Golden template test unchanged (managed path is byte-identical).
• go build ./..., go test -short ./pkg/clouds/aws/codebuild/... ./pkg/cli/client/byoc/aws/..., go vet, gofmt, and golangci-lint on the changed packages all pass (no new findings).

Notes for reviewers

• In adopt mode the RetainBucket parameter + RetainS3Bucket condition are inert (kept defined because the template Metadata/ParameterGroups reference the parameter). Harmless, but could be gated later.
• I couldn't run the //go:build integration TestCloudFormation here (needs live AWS). Worth exercising the adopt path against an account that already has a retained defang-cd-bucket-*.
• Related region-coupling: #2707. Region-precedence bug that made this hard to observe: BYOC AWS: region resolved from AWS_DEFAULT_REGION before AWS_REGION → cross-region cd down hits the wrong region #2191.

Fixes #2192. Re-fixes #358 for the CodeBuild CD (regressed since #422).

Summary by CodeRabbit

• New Features

  • Deployment setup can now reuse eligible retained AWS S3 state buckets.
  • Existing buckets are validated by region and ownership tags before adoption.
  • CloudFormation templates omit creation of a new bucket when an eligible bucket is reused.

• Bug Fixes

  • Ambiguous or incorrectly matched buckets are rejected to prevent unintended adoption.

• Tests

  • Added coverage for bucket discovery, validation, adoption, and rejection scenarios.

[coderabbitai]

Warning

Review limit reached

@defangdevs, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 16 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b51fe615-eba4-4071-a3f6-0cd60e39c520

📥 Commits

Reviewing files that changed from the base of the PR and between b5b1c3c and d129633.

📒 Files selected for processing (6)

• src/cmd/cli/command/cd.go
• src/cmd/cli/command/commands.go
• src/pkg/cli/client/byoc/aws/byoc.go
• src/pkg/cli/client/byoc/do/byoc.go
• src/pkg/cli/client/byoc/gcp/byoc.go
• src/pkg/clouds/aws/codebuild/cfn/setup.go

📝 Walkthrough

Walkthrough

Changes

Retained bucket adoption

Layer / File(s)	Summary
Conditional bucket template generation src/pkg/clouds/aws/codebuild/cfn/template.go, src/pkg/clouds/aws/codebuild/cfn/template_test.go, src/pkg/cli/client/byoc/aws/byoc.go	CreateTemplate accepts an optional existing bucket, omits the managed bucket resource when provided, and reports the selected bucket in outputs. Callers and tests pass the new argument.
Bucket discovery and setup reuse src/pkg/clouds/aws/codebuild/cfn/setup.go, src/pkg/clouds/aws/codebuild/cfn/bucket_test.go	Setup finds buckets matching the prefix, region, and stack tag, rejects ambiguous matches, and reuses a valid bucket. Table-driven tests cover these cases.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AwsCfn.SetUp
  participant S3
  participant findAdoptableBucket
  participant CreateTemplate
  AwsCfn.SetUp->>S3: List CD state buckets
  S3-->>findAdoptableBucket: Return locations and tags
  findAdoptableBucket-->>AwsCfn.SetUp: Return eligible bucket
  AwsCfn.SetUp->>CreateTemplate: Pass existing bucket
  CreateTemplate-->>AwsCfn.SetUp: Return adoption template

Loading

Possibly related PRs

• DefangLabs/defang#1984: Earlier CodeBuild CloudFormation template generation changes that this PR extends.

Suggested reviewers: lionello

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: reusing an existing CD state bucket instead of creating a new one.
Linked Issues check	✅ Passed	The changes implement bucket reuse on re-bootstrap and previous teardown, matching the intent of issues #2192 and #358.
Out of Scope Changes check	✅ Passed	All code changes support bucket adoption and template updates; no unrelated behavior appears introduced.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/pkg/clouds/aws/codebuild/cfn/setup.go`:
- Around line 215-223: Update AwsCfn.SetUp to determine whether the managed
stack already exists before calling findExistingBucket. Only perform
retained-bucket discovery and pass the discovered bucket to CreateTemplate on
the stack-creation path; for existing stacks, preserve the current bucket
configuration in the update template. Add coverage for updating a pre-existing
managed stack.
- Around line 178-183: Normalize the bucketRegion value in the bucket-retention
comparison before checking it against region: map S3’s “EU” LocationConstraint
to “eu-west-1”, while preserving the existing us-east-1 empty-value handling.
Add a test covering an EU response that matches eu-west-1 and confirms the
retained bucket is not duplicated.
- Around line 163-188: Update bucket discovery in setup.go around ListBuckets,
GetBucketLocation, and GetBucketTagging: return any GetBucketLocation error
instead of skipping the bucket, and treat only the S3 NoSuchTagSet error from
GetBucketTagging as a non-match while propagating all other tagging errors. In
bucket_test.go, update mockBucketLister to inject location and tagging errors
and add coverage for both failure paths.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 320d91e0-c165-4fa0-94bf-76d7286cab1a

📥 Commits

Reviewing files that changed from the base of the PR and between 9079c22 and b5b1c3c.

📒 Files selected for processing (5)

• src/pkg/cli/client/byoc/aws/byoc.go
• src/pkg/clouds/aws/codebuild/cfn/bucket_test.go
• src/pkg/clouds/aws/codebuild/cfn/setup.go
• src/pkg/clouds/aws/codebuild/cfn/template.go
• src/pkg/clouds/aws/codebuild/cfn/template_test.go

[defangdevs]

Addressed the review:

• Dropped the automatic ListBuckets discovery (that mirrored reuse cd bucket #422, which was reverted). Adoption is now opt-in only, via a new defang cd install --bucket <name> flag.
• Extended it to all object-storage CD backends: AWS (S3), GCP (GCS), and DigitalOcean (Spaces) each implement SetCDBucket. The command type-asserts the provider, so unsupported providers get a clear error.
• Azure is intentionally excluded: its CD state lives in a Storage Account + Blob Container, not a bucket, so --bucket doesn't map. The flag errors with a message saying so.

The template-level adoption logic (skip the managed Bucket resource, output the given name) is unchanged. build / test -short / vet / gofmt / golangci-lint all pass on the touched packages.

[defangdevs]

Follow-ups pushed:

• DEFANG_CD_BUCKET ⇔ --bucket: AWS and GCP now read DEFANG_CD_BUCKET at construction (DigitalOcean already did), so the env var adopts an existing CD bucket across all commands (up / cd down / …), and --bucket defaults to $DEFANG_CD_BUCKET and overrides it on cd install.
• Azure equivalent (storage account / blob container override) tracked as a backlog issue: Azure: allow overriding the CD storage account / blob container (equivalent of --bucket / DEFANG_CD_BUCKET) #2195.