Label deployed sandbox runtime and portable exports as sandbox

[alexeygrigorev]

Label deployed sandbox runtime and portable exports as sandbox

Status: blocked — completed #166 is historical; prior #146 commit 3491f65acad1986bd002cd1dabbb3e6b8eca0cfa and its BLOCK/ABORT preview are not reusable; #146 waits for #217 deployment and identity verification before fresh reconstruction
Tags: bug, backend, infra, data, testing, P1
Depends on: #217 accepted commit 676a4f66cb4aafa2eef13287bc9c06e6c5f6e115 must pass its HUMAN preview, ordinary OIDC merge/push and deployment, terminal On-Call result, and sanitized HUMAN post-deployment identity verification; #145, #136, and #166 are complete historical prerequisites
Blocks: truthful sandbox runtime/export evidence required by #71 final sandbox acceptance
Next owner: HUMAN/Orchestrator/On-Call complete #217 in its documented sequence; then Software Engineer reconstructs #146 from the exact deployed #217 baseline
Resume condition: #217 records preview PASS from 676a4f66cb4aafa2eef13287bc9c06e6c5f6e115, normal main/OIDC deployment terminal success, and sanitized HUMAN identity proof. Then rebuild #146 from that deployed SHA with fresh lifecycle evidence; never reuse 3491f65 or the old #146 preview.
Evidence: #136 completion, successful ordinary run 31711997388, and current #166 evidence that its repaired chain remains local/unshipped with ordinary deploy temporarily gated.

Authoritative product outcome

DataOps runs only in sandbox for now. With the protected Sponsor migration and its following ordinary sandbox deployment complete, the deployed runtime, portable export manifest, and archive key must identify their source as exact lowercase sandbox, never prod or production.

This is provenance correction, not a resource migration. Preserve every existing stack, logical/physical resource, table, bucket, queue, function, role, domain, key, secret, alarm, retained resource, archive object, and application-data identity. Future production work requires separate user authority and grooming.

Sandbox does not declassify operational data. Sponsor, finance, contact, Telegram, Mailchimp, Trello, execution, export, and restore material retains the existing private-data, encryption, integrity, backup, retention, access, rollback, and public-redaction boundaries.

Current sequencing boundary

The original #145/#136 prerequisite is satisfied. #136 now records the final exact four-GSI Sponsor state, and its one-shot protected migration workflow/migrator was removed under #174. Normal push run 31711997388 subsequently passed checks, OIDC stack deployment, runtime seed, and deployed smoke. No Sponsor migration framework remains for #146 to preserve or test.

#166 is the active sequencing boundary. Its repaired table-cutover chain is local/unshipped, temporarily owns both infra/template.full.yaml and .github/workflows/deploy-dataops-v1.yml, and gates the deploy job to reviewed manual phases until cleanup restores ordinary push deployment. Those are also #146 surfaces, and #146s no-add/remove/replacement preview must compare against the final post-#166 stack—not a transitional phase template.

Eligible now: read-only source/consumer/test inventory and a bounded implementation plan. Not eligible now: editing or freezing the candidate, committing, previewing CloudFormation, merging/pushing, deployment, live environment inspection, or export invocation. After #166 cleanup and one terminal-success ordinary deployment, implement the full candidate in an isolated worktree from that exact stable SHA and repeat all fingerprints/reviews.

#143 is historical context through completed #136, not a #146 execution path. #146 does not reopen Sponsor ownership or authorize Sponsor mutation.

Scope

1. Freeze the stable post-#166 baseline

After the #166 release condition and before editing, record:

• the exact stable DataOps base SHA and clean isolated worktree state;
• the completed Make sandbox the only active DataOps deployment and rollout target #145 source evidence and completed Migrate Sponsor CRM GSIs in a protected stage-only workflow #136/following ordinary-deploy evidence;
• Replace the transitional Tasks table with the final cardId schema #166 A/B/C/D and cleanup acceptance, including removal of temporary phase controls and restored normal push deployment;
• the terminal application stack state from the first ordinary post-cleanup deployment;
• the current stack parameter value for DataOpsEnvironment and the three current DATAOPS_ENV consumers, retained privately where identifiers are sensitive; and
• exact Source, built, packaged, and Processed template fingerprints needed to prove the bounded update.

Public evidence contains only sanitized digests, counts, state, and PASS/FAIL. Do not publish raw stack/resource/account/principal identifiers, full Lambda environment maps, private archive locations, item data, or credentials-adjacent output.

Do not transplant or approve a candidate built against a #166 phase template. Reconstruct it from the final cleanup SHA so the no-delete/no-replacement comparison is authoritative.

2. Make sandbox the only allowed deployed value

In infra/template.full.yaml:

• change DataOpsEnvironment to default exact sandbox;
• restrict allowed values to exact sandbox rather than a free-form pattern; and
• keep DATAOPS_ENV: !Ref DataOpsEnvironment on exactly these existing functions:
  • ConversationalExecutionWorkerFunction;
  • ConversationalResultDispatcherFunction; and
  • WorkEngineFunction.

Do not add the variable to another resource or change any other environment variable.

In .github/workflows/deploy-dataops-v1.yml, add one explicit immutable parameter override:

ParameterKey=DataOpsEnvironment,ParameterValue=sandbox

The explicit override is required so an existing stack parameter cannot remain prod merely because the template default changed. Preserve --config-env full-sandbox, every other parameter/value/order, the existing guard-before-build boundary, and the normal SAM deployment path. No workflow input, variable, secret, matrix, branch, environment, or caller may select another value.

Update the final ordinary deployment-contract tests after #166 cleanup to accept only this one parameter addition. Preserve the final Sponsor ownership guard and the absence of migration machinery; do not reintroduce a migrator or any superseded application-deploy mechanism.

3. Preserve portable export and archive contracts

The deployed export path already passes DATAOPS_ENV through handler/cron archive generation into portable export metadata. Keep the existing schema and path algorithms, and prove:

• every deployed portable export has manifest.json.source_environment == "sandbox";
• scheduled and manually routed deployed archive generation use that same value;
• new archive keys are exactly <existing-prefix>/sandbox/<YYYY-MM-DD>/dataops-execution-<timestamp>.tar.gz;
• archive bucket, prefix parameter, encryption, versioning, retention, checksum, manifest schema/version, and object-writing behavior are otherwise unchanged; and
• missing/local/test environment behavior remains explicit and cannot silently label a deployed export as sandbox without the stack parameter.

Do not rename or move existing archive objects under historical prod or other prefixes. Do not rewrite historical manifests or evidence. The backward-compatible restore-evidence field production_write_gate remains a generic no-production-write safety field; it is not source-environment provenance and must continue to prohibit production writes. Restore drill target_environment remains an explicit isolated non-production target and is not forced to the active sandbox name.

Update current public examples/documentation, including docs/v1-execution-data-safety.md and docs/restore-drill.md, to show sandbox as the active source environment while retaining generic future-production prohibitions and historical-data safety.

4. No-delete/no-replacement preview

Before the accepted commit is merged/pushed for ordinary deployment, an authorized HUMAN sandbox operator must create one exact non-executing CloudFormation change-set preview from the frozen packaged candidate and terminal post-#166-cleanup stack baseline.

The preview may contain only in-place, non-replacing configuration changes to the three functions above caused by DATAOPS_ENV: sandbox, plus the stack parameter value. It must contain:

• zero Remove actions;
• zero Add actions;
• zero replacements, conditional replacements, deletes, imports, or resource-policy/retention changes;
• zero table, bucket, queue, role, policy, key, secret, domain, API, event-source, schedule, alarm, log-group, or retained-resource changes; and
• no application flag/provider/credential change.

Any additional resource or ambiguous replacement evaluation is BLOCK/ABORT and returns to Architecture. Never execute the preview. After evidence is retained, delete only that exact unexecuted preview and prove it absent. The accepted application commit then deploys only through the normal main GitHub Actions workflow; the preview is not a manual app deploy and cannot substitute for CI/CD.

Expected changed surfaces

After #166 cleanup, the candidate should be limited to:

• infra/template.full.yaml;
• .github/workflows/deploy-dataops-v1.yml;
• tests/infra/test_template.py and the final ordinary-deployment contract test present after Replace the transitional Tasks table with the final cardId schema #166 cleanup;
• focused shipped portable-export/archive backend tests; and
• docs/v1-execution-data-safety.md plus docs/restore-drill.md.

scripts/deploy/sponsor-crm-gsi-core.test.mjs no longer exists and must not be restored, replaced, or run: the Sponsor migration was one-shot work deleted after completion. No #136/#166 phase test becomes permanent #146 coverage. Backend production source should not change unless Tester demonstrates that existing DATAOPS_ENV propagation cannot meet the exact criteria. Any source change must remain a pure provenance fix with no export entity/schema, data selection, archive-write, restore, or authorization behavior change.

Acceptance criteria

• Historical dependencies are satisfied: Make sandbox the only active DataOps deployment and rollout target #145 source is integrated, Migrate Sponsor CRM GSIs in a protected stage-only workflow #136 records the exact four-GSI final state and removal of its one-shot migration machinery, and following ordinary run 31711997388 reached terminal success; no pre-Migrate Sponsor CRM GSIs in a protected stage-only workflow #136 Label deployed sandbox runtime and portable exports as sandbox #146 implementation or deployment occurred.
• Make SAM artifacts and scheduled-target previews identity-stable #217 completes its HUMAN preview, normal main/OIDC deployment, terminal On-Call result, and HUMAN post-deployment identity verification before a fresh Label deployed sandbox runtime and portable exports as sandbox #146 candidate or preview.
• DataOpsEnvironment defaults to and permits only exact lowercase sandbox.
• Normal deployment explicitly passes DataOpsEnvironment=sandbox; no workflow input/configuration can select prod, production, or another environment.
• Exactly the three existing functions receive DATAOPS_ENV=sandbox; no other environment variable or function/resource configuration changes.
• Ordinary deployment retains full-sandbox, guard placement, all other parameter values/order, and the existing SAM/CI path; the final Sponsor ownership guard and completed absence of migration machinery are unchanged.
• Deployed portable manifests report exact source_environment: sandbox through both manual-route and scheduled archive paths.
• New archive keys use the existing prefix followed by exact /sandbox/ and the existing date/filename format; bucket, encryption, versioning, retention, checksums, schemas, and write behavior remain unchanged.
• Historical archives/manifests/evidence are not moved, renamed, relabeled, deleted, or rewritten.
• Restore evidence still rejects production targets and performs no restore/import/write; its backward-compatible safety field remains distinct from source-environment provenance.
• Current docs show sandbox source provenance without weakening confidentiality, backup, restore, rollback, or future-production prohibitions.
• A parsed Source/Processed-template structural diff proves no logical resource, identity, policy, retention control, event, schema, table property, bucket property, or non-environment configuration change.
• Architecture accepts the frozen source/build/template diff and exact no-delete/no-replacement preview contract.
• Security accepts sandbox-only parameter closure, export/archive confidentiality, sanitized evidence, and the absence of credential/provider/permission changes.
• Tester runs the complete workflow below with exact exit codes/counts and verifies every criterion. No UI changes are expected, so screenshots are not required.
• PM accepts operator-visible provenance, documentation, archive semantics, and preservation of historical evidence/resource identities.
• Software Engineer reconstructs from the exact deployed Make SAM artifacts and scheduled-target previews identity-stable #217 baseline, completes fresh Architecture/Security/Tester/PM gates, and commits the newly accepted candidate with Refs Label deployed sandbox runtime and portable exports as sandbox #146; never reuse 3491f65 or its preview.
• [HUMAN] An authorized sandbox operator creates, reviews, and deletes the exact non-executing change-set preview; sanitized evidence proves zero add/remove/replacement and only the three expected in-place function configuration changes.
• On-Call monitors normal CI/CD, verifies the exact accepted SHA reaches terminal stack success, and records a sanitized read-only deployed parameter/function check without exposing full environment maps.
• [HUMAN] An authorized sandbox operator invokes one bounded existing export/archive path, retains the archive privately, and verifies exact sandbox manifest/key provenance, validation/checksum success, and unchanged confidentiality. No item bodies or archive URI are posted publicly.
• No production deployment/data action, manual application deployment, Sponsor migration, provider call, Telegram action, feature enablement, queue action, restore/import, cleanup, or historical-evidence rewrite occurs under Label deployed sandbox runtime and portable exports as sandbox #146.

Required automated verification

npm --prefix backend test
npm --prefix backend run typecheck
npm --prefix backend run build
uv run --with pytest python -m pytest -q tests/infra
make sam-validate
make sam-build
git diff --check <accepted-base>...HEAD

Focused tests must additionally prove:

• template default/allowed-values and explicit workflow override are exact;
• removal, alternate value, duplicate override, caller-selectable input, or prod/production substitution fails;
• only the three named functions consume the parameter;
• deployed handler and scheduled cron archive routes produce source_environment=sandbox and /sandbox/ keys;
• missing/local/test values remain distinguishable;
• historical/non-production restore target rejection and archive-key sanitization tests remain valid;
• Source and built template resource inventories/identities are equal to baseline; and
• the only expected Processed resource-property delta is DATAOPS_ENV on the three functions.

Run only focused shipped portable-export/archive validation against synthetic/local data. Do not run one-off import, export-source, migration, or restore scripts as ongoing verification. Browser E2E and screenshots are not required because no UI route, page, or layout changes.

Test scenarios

Existing stack retains the old parameter value unless explicitly overridden

Given the terminal post-#166-cleanup stack previously used prod
When the new normal deployment runs
Then the explicit parameter override changes it to exact sandbox; relying on a new default alone is not accepted.

Manual and scheduled exports agree

Given the deployed WorkEngine function has DATAOPS_ENV=sandbox
When the authenticated export route and scheduled export/archive route run
Then each manifest says sandbox and each new archive key uses the same sandbox segment.

Alternate environment is attempted

Given a template, workflow, input, parameter, or ambient value tries prod, production, staging, or any non-sandbox deployed value
When validation/build/tests run
Then the candidate fails before deployment.

CloudFormation preview contains unexpected impact

Given the frozen post-#166-cleanup baseline and packaged #146 candidate
When the non-executing change set is reviewed
Then only three non-replacing function configuration modifications are accepted; any add/remove/replacement or unrelated modification aborts.

Historical archive remains historically accurate

Given an existing archive whose manifest/path records an older environment value
When #146 deploys
Then that object and manifest remain unchanged; only new exports use sandbox provenance.

Real sandbox export evidence remains private

Given an authorized bounded sandbox export after deployment
When provenance and checksum are verified
Then only sanitized PASS/FAIL, counts, digests, and UTC time are public; item data, full environment maps, identities, and archive locations remain private.

Role and release sequence

0. Now: PM/Software Engineer may perform read-only source, consumer, test, and conflict inventory only. No candidate edit or fingerprint is authoritative during Replace the transitional Tasks table with the final cardId schema #166.
1. Replace the transitional Tasks table with the final cardId schema #166 completes repaired A/B/C/D and cleanup, restores ordinary push-triggered OIDC deployment, and records one terminal-success normal main run.
2. Software Engineer creates a clean isolated worktree from that exact stable SHA, implements the complete Label deployed sandbox runtime and portable exports as sandbox #146 candidate without commit, and freezes exact source/build/packaged/Processed/diff fingerprints.
3. Architecture reviews template/resource identity and the exact no-add/remove/replacement preview contract.
4. Security reviews sandbox-only parameter closure, data confidentiality, and evidence redaction.
5. Tester runs only the ordinary product/infra verification listed here; no retired migration or one-off import/restore suite.
6. Product Manager accepts the immutable candidate.
7. Software Engineer commits the accepted candidate with Refs #146; do not merge or push yet.
8. [HUMAN] An authorized sandbox operator creates, reviews, and deletes the exact non-executing preview from that immutable commit. Any action beyond the three expected in-place function configuration modifications is BLOCK/ABORT.
9. After preview PASS, the orchestrator merges/pushes through normal main; On-Call monitors the ordinary OIDC deployment and performs sanitized read-only parameter/function verification.
10. [HUMAN] An authorized sandbox operator invokes one bounded existing product export/archive path and privately verifies manifest/key/checksum provenance. PM may close only after this evidence passes.

Out of scope

• Editing/freezing a candidate, CloudFormation preview, merge/push, deploy, or runtime/export evidence before Replace the transitional Tasks table with the final cardId schema #166 cleanup restores and proves the stable ordinary deployment path. Read-only planning inventory is allowed.
• Production environments, parameters, deployments, data mutation, smoke, rollout, or future-production design.
• Renaming or replacing legacy stacks/resources/domains or changing retention/deletion policies.
• Sponsor ownership/GSI work, Reconcile Sponsor CRM table ownership with CloudFormation #143 evidence, queue reconciliation, provider/Telegram behavior, feature flags, credentials, IAM/OIDC, or GitHub environment configuration.
• Export entity/schema changes, importing/restoring data, moving historical archives, changing archive bucket/prefix/retention/encryption, or destructive cleanup.
• Manual application deployment or executing the preview change set.
• Private evidence, operational data, identifiers, paths, links, or credentials in the public repository or tracker.

[alexeygrigorev]

PM GROOMED — post-#136 sandbox provenance correction is agent-ready but blocked

#146 is deliberately sequenced after #145 and the complete #136 migration plus ordinary sandbox deploy. This protects #136’s accepted template/deploy contract and starts #146 from the terminal post-migration stack baseline.

The implementation is bounded to exact DataOpsEnvironment=sandbox closure, the three existing DATAOPS_ENV consumers, portable manifest/archive provenance, focused docs/tests, and one explicit normal-deploy parameter. Legacy resource identities, historical archives, export schemas, Sponsor ownership/migration logic, confidentiality, and normal CI/CD remain unchanged.

A separately approved HUMAN no-execute change-set preview must prove zero add/remove/replacement and only the three expected in-place function configuration changes. The immutable candidate is committed with Refs #146, previewed without execution, and merged/pushed only after preview PASS. Normal GitHub Actions remains the sole app deploy.

needs grooming is removed. Current status remains blocked. Exact resume signal: #145 is complete/effective and #136 records On-Call PASS for exact-prefix-four migration followed by terminal-success ordinary sandbox deployment. Exact next role after that signal: Software Engineer, followed by Architecture, Security, Tester, and PM.

[alexeygrigorev]

PM DEPENDENCY RECONCILIATION — #145 complete; #136 terminal success still required

#145 is now a completed source prerequisite at d0c2b5033d3272c7eda9db61868cd106742e2bb0, with checks PASS and zero migration dispatches.

#146 remains blocked. The failed #145 automatic deploy is not its resume signal. It still requires #136 migration-only exact-prefix-four On-Call PASS followed by an ordinary deploy-dataops-v1 terminal-success sandbox deployment on the accepted SHA. No #146 implementation, deploy, export, or HUMAN criterion advances now.

[alexeygrigorev]

PM lifecycle reconciliation — historical gate complete; #166 is the active sequencing boundary

The original #145/#136 hold is satisfied:

• Make sandbox the only active DataOps deployment and rollout target #145 is complete at d0c2b5033d3272c7eda9db61868cd106742e2bb0.
• #136 records completion: the Sponsor table is ACTIVE with the exact four required GSIs ACTIVE, and the one-shot migration workflow/migrator was deleted under Audit one-shot migration tooling and remove it from the Lambda artifact #174.
• The following ordinary push run 31711997388 at 794076354048f6ff3417d55d13d334caf4237cea passed checks, OIDC stack deployment, runtime seed, and deployed smoke.

#146 does not start from that now-historical SHA because #166 currently owns the same application template and deploy workflow. Its repaired A/B/C/D chain is still local/unshipped, the deploy job is temporarily manual-phase-only, and cleanup must restore ordinary push deployment. Freezing or previewing #146 against a transitional #166 template would invalidate the promised zero-add/remove/replacement comparison.

Exact phase decision

1. Eligible now, agent/read-only: source consumer inventory, focused shipped-test mapping, and conflict planning only. No candidate bytes or fingerprints are authoritative yet.
2. Implementation resume: Replace the transitional Tasks table with the final cardId schema #166 records accepted A/B/C/D outcomes, removes temporary phase controls, restores ordinary push-triggered OIDC deployment, and one following normal main run reaches terminal stack success. Then implement the complete Label deployed sandbox runtime and portable exports as sandbox #146 candidate in an isolated worktree from that exact stable SHA.
3. Agent gates: Architecture, Security, Tester, PM, then an explicit-path commit with Refs #146. Retired Sponsor migration tests and one-off import/export/restore scripts are not ongoing verification.
4. HUMAN preview gate: only after the immutable accepted commit, create/review/delete one non-executing change set. Accept only the three expected non-replacing function configuration modifications; any add/remove/replacement or unrelated resource change is BLOCK/ABORT.
5. Release/runtime gates: merge/push through restored normal CI; On-Call verifies the deployed parameter and three consumers using sanitized read-only evidence; then a HUMAN invokes one bounded existing product export/archive path and privately verifies sandbox manifest/key/checksum provenance.

The issue body now reflects this sequence, checks the completed historical dependency criterion, adds the explicit #166 cleanup criterion, removes the deleted scripts/deploy/sponsor-crm-gsi-core.test.mjs from expected coverage, and removes one-off dry-run import from ongoing verification. #146 remains open and continues to block truthful sandbox provenance evidence for #71.

No repository edit, test, AWS/IAM call, preview, workflow dispatch, credential/provider action, export/import/restore, or data operation was performed.