fix(purge): physical account-purge hardening — expert reserves #47 + REAL GCS/k8s E2E [PROVEN_REVIEW_PENDING] - #52
Open
openaxcloud wants to merge 26 commits into
Open
fix(purge): physical account-purge hardening — expert reserves #47 + REAL GCS/k8s E2E [PROVEN_REVIEW_PENDING]#52openaxcloud wants to merge 26 commits into
openaxcloud wants to merge 26 commits into
Conversation
Contributor
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 035ee9ed27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+440
to
+444
| const collaborations = await this.prisma.projectCollaborator.findMany({ | ||
| where: { userId }, | ||
| select: { projectId: true }, | ||
| }); | ||
| const workspaceProjectIds = [...new Set([...bucketProjectIds, ...collaborations.map((c) => c.projectId)])]; |
There was a problem hiding this comment.
An organization member can open any project in that organization without receiving a ProjectCollaborator row (requireProject explicitly authorizes via org membership in app.ts), but this inventory includes shared projects only through projectCollaborator. Consequently, a user who opened a project as a normal member of a multi-member organization has a deterministic per-user PVC that is omitted from physical erasure, while the account can still be stamped purged with a zero-remaining proof.
Useful? React with 👍 / 👎.
Comment on lines
+1138
to
+1140
| const pvc = await this.k8s.get('PersistentVolumeClaim', namespace, pvcName).catch(() => undefined); | ||
|
|
||
| return Boolean(pvc); |
There was a problem hiding this comment.
When the Kubernetes API is unavailable, RBAC is misconfigured, or get otherwise fails, this catch converts the error into undefined, making pvcExists report false. The purge then treats the PVC as already absent, skips deletion, and can persist a verified erasure proof even though the live volume was never checked; only a genuine not-found response should map to false, while operational errors must fail closed.
Useful? React with 👍 / 👎.
Comment on lines
+1156
to
+1160
| await Promise.allSettled([ | ||
| this.k8s.delete('Secret', namespace, workspace.agentTokenSecretName ?? `agent-token-${workspaceId}`), | ||
| this.k8s.delete('Pod', namespace, workspace.podName), | ||
| this.k8s.delete('Service', namespace, workspace.serviceName), | ||
| ]); |
There was a problem hiding this comment.
If deleting the token Secret, Pod, or Service fails, Promise.allSettled resolves successfully and none of its rejected results are inspected, so the API is told that the write barrier held. Because bucket deletion runs before workspace deletion, a surviving pod can write or recreate object storage after its bucket was verified and before it is stopped, allowing the purge to certify a transient zero state rather than durable erasure.
Useful? React with 👍 / 👎.
Comment on lines
+27135
to
+27138
| { | ||
| objectStorage: resolveObjectStorage(), | ||
| workspaceVolumes: createWorkspaceVolumeEraser(), | ||
| writeBarrier: createWriteBarrier((inv) => inv.workspaceIds), |
There was a problem hiding this comment.
When OBJECT_STORAGE_ENABLED is unset or deliberately disabled during the documented rollback procedure, resolveObjectStorage() returns NoopObjectStorage, whose bucketExists() always returns false. Passing it here makes every inventory bucket appear absent and permits a verified purge even though buckets created before the flag was disabled remain in GCS; destructive erasure must use the real backend independently of the user-facing feature switch or fail closed when the adapter is inactive.
AGENTS.md reference: AGENTS.md:L3-L5
Useful? React with 👍 / 👎.
Comment on lines
+27136
to
+27138
| objectStorage: resolveObjectStorage(), | ||
| workspaceVolumes: createWorkspaceVolumeEraser(), | ||
| writeBarrier: createWriteBarrier((inv) => inv.workspaceIds), |
There was a problem hiding this comment.
The production write barrier passed here only freezes workspace-manager resources; it does not block the API's object-storage routes or revoke project-scoped object-storage grants, which are valid for up to seven days and bypass user-session authentication. A holder can therefore recreate a bucket after eraseBucket performs its live zero check but before the database tombstone is committed, leaving physical data behind despite a verified proof; the barrier must reject storage mutations for the purging subject/projects before deletion begins.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR hardens the “physical account purge” path by introducing a write barrier and “real disappearance” verification (live GCS + live Kubernetes PVC existence), expanding the purge inventory to cover all projects a data subject touched, and adding replayable E2E evidence artifacts.
Changes:
- Add workspace-manager control-plane endpoints to (1) freeze a workspace (write barrier) and (2) check real PVC existence via live Kubernetes.
- Implement/store-layer purge executor wiring (API + Prisma + in-memory test store) with fail-closed physical erasure gating and structured erasure proofs.
- Add unit, route, and DB-backed tests plus replayable GCS/kind E2E proof scripts and committed evidence artifacts.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| services/workspace-manager/src/manager.ts | Adds pvcExists and freezeWorkspace primitives used by the purge write barrier + live PVC verification. |
| services/workspace-manager/src/app.ts | Exposes authenticated manager routes for pvc-exists and freeze. |
| services/worker/src/index.ts | Adds account.purge worker trigger that POSTs to the API’s internal purge executor. |
| services/api/src/tests/test-api-store.ts | Implements in-memory mirror of purgeUserAccount including physical-erasure gating and proof creation for tests. |
| services/api/src/tests/account-storage-purge.spec.ts | Unit tests for eraseSubjectStorage behavior (write barrier ordering, fail-closed semantics, inventory coverage). |
| services/api/src/tests/account-purge-routes.spec.ts | Route-level tests for /internal/account-purge behavior (auth, dry-run default, idempotence, concurrency, fail-closed). |
| services/api/src/tests/account-purge-db.spec.ts | DB-backed durable proofs for purge semantics (real Postgres, advisory lock concurrency, ledger immutability). |
| services/api/src/store.ts | Extends ApiStore contract with purgeUserAccount. |
| services/api/src/prisma-store.ts | Implements production purge executor with advisory-lock serialization, per-class deletes/anonymization/retention, and verification. |
| services/api/src/app.ts | Wires /internal/account-purge route + default physical-erasure adapters (workspace-manager + object storage) and write barrier. |
| services/api/src/account-storage-purge.ts | Introduces physical storage erasure orchestrator with ports, evidence, and verification. |
| services/api/src/account-purge.ts | Defines erasure-proof types, helpers, and proof verification rules. |
| services/api/scripts/physical-purge-k8s-e2e.sh | Replayable kind-based PVC deletion E2E proof script with hashed artifact output. |
| services/api/scripts/physical-purge-gcs-e2e.ts | Replayable GCS adapter E2E proof script that seeds/list/deletes/verifies and writes hashed artifacts. |
| infra/helm/platform/templates/cronjobs.yaml | Schedules the account.purge enqueue CronJob (same pattern as other enterprise jobs). |
| docs/deploy-evidence/2026-07-23-physical-purge-e2e/README.md | Documents the real GCS + Kubernetes proof artifacts and replay steps. |
| docs/deploy-evidence/2026-07-23-physical-purge-e2e/k8s-SHA256SUMS | Hash for the Kubernetes E2E artifact. |
| docs/deploy-evidence/2026-07-23-physical-purge-e2e/k8s-proof.json | Kubernetes E2E proof artifact (before/after, verified true). |
| docs/deploy-evidence/2026-07-23-physical-purge-e2e/gcs-SHA256SUMS | Hash for the GCS E2E artifact. |
| docs/deploy-evidence/2026-07-23-physical-purge-e2e/gcs-proof.json | GCS E2E proof artifact (before/after, classes, verified true). |
| docs/deploy-evidence/2026-07-22-account-purge-worker/README.md | Background + delivery summary for the purge worker/executor and its proofs. |
| docs/deploy-evidence/2026-07-22-account-purge-worker/proof-sample.json | Sample proof artifact intended to show a complete erasure proof shape. |
| docs/deploy-evidence/2026-07-22-account-purge-worker/JOURNAL.md | Repro journal for the Postgres-backed proof run and helm render validation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1135
to
+1141
| async pvcExists(namespace: string, workspaceId: string): Promise<boolean> { | ||
| const workspace = await this.store.get(workspaceId).catch(() => undefined); | ||
| const pvcName = workspace?.pvcName ?? `pvc-${workspaceId}`; | ||
| const pvc = await this.k8s.get('PersistentVolumeClaim', namespace, pvcName).catch(() => undefined); | ||
|
|
||
| return Boolean(pvc); | ||
| } |
Comment on lines
+1149
to
+1164
| async freezeWorkspace(namespace: string, workspaceId: string): Promise<void> { | ||
| const workspace = await this.store.get(workspaceId).catch(() => undefined); | ||
|
|
||
| if (!workspace) { | ||
| return; | ||
| } | ||
|
|
||
| await Promise.allSettled([ | ||
| this.k8s.delete('Secret', namespace, workspace.agentTokenSecretName ?? `agent-token-${workspaceId}`), | ||
| this.k8s.delete('Pod', namespace, workspace.podName), | ||
| this.k8s.delete('Service', namespace, workspace.serviceName), | ||
| ]); | ||
|
|
||
| this.lastTouchAt.delete(workspaceId); | ||
| await this.store.update(workspaceId, { status: 'STOPPED' }).catch(() => {}); | ||
| } |
Comment on lines
+1
to
+5
| { + | ||
| "kind": "account-erasure-proof", + | ||
| "userId": "cmrvpdz4t000fi0lnerebnm5i", + | ||
| "classes": [ + | ||
| { + |
openaxcloud
pushed a commit
that referenced
this pull request
Jul 23, 2026
…ED (2 signés), 4 refusés + 2 contrats refusés Reçu -07 COMPLET (réponse brute incoming/, sha256 f5771529…, commits audités §1, décision machine §9). SIGNED/CLOSED reviewer OpenAI-Codex UNIQUEMENT : - P0-LS-13 (session navigateur fail-closed, observations reliées au run) - P0-LS-03 (couverture de hash du paquet, job CI officiel) avec limites de portée verbatim. Rien d'autre. REFUSED (motifs verbatim) : P0-LS-16/LS-18/V3-14 — le vérificateur v6 reste FAIL-OPEN sur l'ABSENCE de mergedCommit/repoCommit/runUrl (contrôlés seulement s'ils existent) ; correction v7 exigée (champs obligatoires non vides + négatifs par suppression + roll post-merge). P0-A2-09 — repro.sh pose le trap teardown APRÈS create+billing (fuite projet possible). Contrats REFUSED_V4, reviewer NON écrit : CTR-OPERATIONS-DR (obligations BLOCKED/UNTESTED ; sous-artefacts = preuves individuelles) + CTR-RUNTIME-NIX (négatif live non exécuté). PR #39 ACCEPTÉE À PORTÉE CIBLÉE — AUCUNE signature de CTR-BILLING-LEDGER ni preuve de prod déduite. PR #51/#52 refusées. Compteurs : 30 CLOSED / 27 OPEN / 5 PROVEN / 3 PROVEN_REVIEW_PENDING = 65. IMPLEMENTATION_STATUS + vues de suivi régénérées, validateur vert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Contributor
openaxcloud
pushed a commit
that referenced
this pull request
Jul 31, 2026
…SIGNÉS (33 CLOSED), refus verbatim inscrits, 2 sous-preuves à portée limitée Reçu -08 COMPLET : responseHash RECOMPUTABLE bcdd1d20… du fichier committé ; hash annoncé par l'expert d1e99781… consigné avec l'écart DÉCLARÉ (le fichier contient sa propre ligne de hash — auto-match impossible par construction). - SIGNÉS reviewer OpenAI-Codex : P0-LS-16, P0-LS-18, P0-V3-14 → CLOSED. 33 CLOSED / 27 OPEN / 5 PROVEN = 65 (0 PROVEN_REVIEW_PENDING). - REFUSÉS verbatim : P0-A2-09 (teardown describe traité comme absence — reste OPEN/REFUSED/UNKNOWN) ; lots purge #51 (ChatShare public survivant, audit ciblant l'utilisateur) + #52 (route thumbnail hors barrière, topologie non sérialisée) consignés au reçu ; CTR-RUNTIME-NIX refus v5 (code typé non capturé, référence .log morte, sur-revendication UI). - CTR-OPERATIONS-DR : AUCUN reviewer (non soumis en signature entière). - 2 sous-preuves acceptées à PORTÉE LIMITÉE, limites verbatim : EVID-DR-SNAPSHOT-001 (pas de couverture auto des futurs disques ; archiver JSON brute describe + Audit Logs) et NIX-REVOKED-GENERATION-FAILED-410-AND-RESTORE-READY-200. Vues régénérées par script ; validateur vert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Jul 31, 2026
…ze barrier + topology-drift guard RR-08 refused PR #52 on two paths; both fixed fail-closed with executable tests. 1) Object-storage write barrier was bypassable. POST /projects/:id/thumbnail/ upload-url called ensureBucket + createUploadUrl with NO freeze check, so it (and the background thumbnail capturer, and any future signed-upload route) could recreate a bucket/object AFTER the purge zero-check. Fixed structurally: - guardObjectStorageWrites() wraps the storage so every CREATE/MODIFY primitive (ensureBucket/createUploadUrl/putObject/moveObject) REFUSES a frozen project with OBJECT_STORAGE_PURGE_FROZEN (→ 403). resolveObjectStorage() returns the guarded wrapper for ALL routes + the capturer; the purge's own erasure uses the RAW adapter (resolveRawObjectStorage) so it can still delete the project it froze. - the thumbnail route also calls the explicit objectStorageWriteBlocked guard. Tests: tests/object-storage-purge-freeze.spec.ts (thumbnail→403, upload-url→403, reads still 200, unfreeze→200); object-storage.spec.ts guardObjectStorageWrites unit (writes refused, reads/deletes pass, unfrozen project writes). 2) Topology wasn't serialized against the tombstone. The external GCS/PVC erasure ran on the PRE-transaction sole/shared topology; the tx recomputed it independently, so a membership race (shared→sole / sole→shared) during erasure could strand a newly-sole org's bucket or destroy a newly-shared org's bucket yet still stamp purgedAt. Fixed: resolveStorageTopology() computes a stable fingerprint; the purge tx re-derives it under the advisory lock and THROWS ACCOUNT_PURGE_TOPOLOGY_DRIFT before any delete/tombstone on drift — account stays queued, next run recomputes fresh (idempotent erasure). Never finalize on a stale inventory. Tests (real Postgres): tests/account-purge-db.spec.ts (6) shared→sole and (7) sole→shared — mutate membership inside the eraseStorage hook (the real race window) and assert abort + no tombstone + intact storage rows. Root typecheck + CI-strict build (tsc src/server.ts) clean; 76 affected api tests + 6 real-PG purge tests + 28 object-storage tests green. PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Jul 31, 2026
…revious synchronize) No code change; forces a fresh pull_request synchronize so Production CI / PR Validation / Code Quality / Security / Preview run on the RR-08 fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
force-pushed
the
vague3-purge47-hardening
branch
from
July 31, 2026 16:04
795932a to
2627afd
Compare
openaxcloud
pushed a commit
that referenced
this pull request
Jul 31, 2026
…ze barrier + topology-drift guard RR-08 refused PR #52 on two paths; both fixed fail-closed with executable tests. 1) Object-storage write barrier was bypassable. POST /projects/:id/thumbnail/ upload-url called ensureBucket + createUploadUrl with NO freeze check, so it (and the background thumbnail capturer, and any future signed-upload route) could recreate a bucket/object AFTER the purge zero-check. Fixed structurally: - guardObjectStorageWrites() wraps the storage so every CREATE/MODIFY primitive (ensureBucket/createUploadUrl/putObject/moveObject) REFUSES a frozen project with OBJECT_STORAGE_PURGE_FROZEN (→ 403). resolveObjectStorage() returns the guarded wrapper for ALL routes + the capturer; the purge's own erasure uses the RAW adapter (resolveRawObjectStorage) so it can still delete the project it froze. - the thumbnail route also calls the explicit objectStorageWriteBlocked guard. Tests: tests/object-storage-purge-freeze.spec.ts (thumbnail→403, upload-url→403, reads still 200, unfreeze→200); object-storage.spec.ts guardObjectStorageWrites unit (writes refused, reads/deletes pass, unfrozen project writes). 2) Topology wasn't serialized against the tombstone. The external GCS/PVC erasure ran on the PRE-transaction sole/shared topology; the tx recomputed it independently, so a membership race (shared→sole / sole→shared) during erasure could strand a newly-sole org's bucket or destroy a newly-shared org's bucket yet still stamp purgedAt. Fixed: resolveStorageTopology() computes a stable fingerprint; the purge tx re-derives it under the advisory lock and THROWS ACCOUNT_PURGE_TOPOLOGY_DRIFT before any delete/tombstone on drift — account stays queued, next run recomputes fresh (idempotent erasure). Never finalize on a stale inventory. Tests (real Postgres): tests/account-purge-db.spec.ts (6) shared→sole and (7) sole→shared — mutate membership inside the eraseStorage hook (the real race window) and assert abort + no tombstone + intact storage rows. Root typecheck + CI-strict build (tsc src/server.ts) clean; 76 affected api tests + 6 real-PG purge tests + 28 object-storage tests green. PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Jul 31, 2026
…revious synchronize) No code change; forces a fresh pull_request synchronize so Production CI / PR Validation / Code Quality / Security / Preview run on the RR-08 fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
openaxcloud
pushed a commit
that referenced
this pull request
Aug 3, 2026
… portée ciblée ; A2-09/PR #52/CTR-RUNTIME-NIX refusés (motifs verbatim), aucun nouveau SIGNED/CLOSED Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
openaxcloud
force-pushed
the
vague3-purge47-hardening
branch
from
August 3, 2026 07:01
2627afd to
886522b
Compare
openaxcloud
pushed a commit
that referenced
this pull request
Aug 3, 2026
…ze barrier + topology-drift guard RR-08 refused PR #52 on two paths; both fixed fail-closed with executable tests. 1) Object-storage write barrier was bypassable. POST /projects/:id/thumbnail/ upload-url called ensureBucket + createUploadUrl with NO freeze check, so it (and the background thumbnail capturer, and any future signed-upload route) could recreate a bucket/object AFTER the purge zero-check. Fixed structurally: - guardObjectStorageWrites() wraps the storage so every CREATE/MODIFY primitive (ensureBucket/createUploadUrl/putObject/moveObject) REFUSES a frozen project with OBJECT_STORAGE_PURGE_FROZEN (→ 403). resolveObjectStorage() returns the guarded wrapper for ALL routes + the capturer; the purge's own erasure uses the RAW adapter (resolveRawObjectStorage) so it can still delete the project it froze. - the thumbnail route also calls the explicit objectStorageWriteBlocked guard. Tests: tests/object-storage-purge-freeze.spec.ts (thumbnail→403, upload-url→403, reads still 200, unfreeze→200); object-storage.spec.ts guardObjectStorageWrites unit (writes refused, reads/deletes pass, unfrozen project writes). 2) Topology wasn't serialized against the tombstone. The external GCS/PVC erasure ran on the PRE-transaction sole/shared topology; the tx recomputed it independently, so a membership race (shared→sole / sole→shared) during erasure could strand a newly-sole org's bucket or destroy a newly-shared org's bucket yet still stamp purgedAt. Fixed: resolveStorageTopology() computes a stable fingerprint; the purge tx re-derives it under the advisory lock and THROWS ACCOUNT_PURGE_TOPOLOGY_DRIFT before any delete/tombstone on drift — account stays queued, next run recomputes fresh (idempotent erasure). Never finalize on a stale inventory. Tests (real Postgres): tests/account-purge-db.spec.ts (6) shared→sole and (7) sole→shared — mutate membership inside the eraseStorage hook (the real race window) and assert abort + no tombstone + intact storage rows. Root typecheck + CI-strict build (tsc src/server.ts) clean; 76 affected api tests + 6 real-PG purge tests + 28 object-storage tests green. PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Aug 3, 2026
…revious synchronize) No code change; forces a fresh pull_request synchronize so Production CI / PR Validation / Code Quality / Security / Preview run on the RR-08 fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Aug 3, 2026
…e erasure RR-09 refused #52: the RR-08 drift guard fired AFTER the irreversible GCS/PVC deletion, so in sole→shared the bucket was already destroyed (the guard only blocked the tombstone); and the object-storage freeze was never released, leaving a residual freeze on abort. Reordered + hardened, fail-closed: (1) GUARANTEE BEFORE DELETION — acquirePurgeGuarantee() runs first, in one tx under the per-user advisory lock: computes the authoritative sole/shared topology AND freezes it (membership for every org the subject belongs to + object storage for the sole-org buckets) atomically, records a recoverable plan. The erasure then runs on THIS locked inventory only. (2) MEMBERSHIP BLOCKED DURING ERASURE — addMember/removeMember take the freeze-set advisory lock and refuse (MEMBERSHIP_FROZEN_FOR_PURGE) for a frozen org, so no join/leave can flip sole↔shared mid-erasure. (3) DELETE ONLY AFTER THE GUARANTEE — eraseStorage is invoked only once a guarantee is held, on guarantee.bucketProjectIds. (4) RECOVERABLE STATE MACHINE — releasePurgeGuarantee() runs in a finally on EVERY exit (purged/drift/throw): unfreeze membership + object storage, clear the plan. reconcilePurgeFreezes() (run at purge-executor start) releases a plan left by a crashed run — no residual freeze. Object-storage freeze moved out of createWriteBarrier into the store guarantee; the barrier now only freezes pods. Tests (real Postgres, account-purge-db.spec.ts): (6) shared bucket never handed to the erasure + survives; (7) co-member cannot leave while frozen; (8) new member cannot join while frozen; (9) NO residual freeze after a failed purge + org writable again; (10) reconciler releases a crashed run's freeze. RR-08 drift check kept as a defence-in-depth backstop. Root typecheck (all services) + CI-strict build clean; 85 affected api tests green (incl. 9 real-PG purge). PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
openaxcloud
force-pushed
the
vague3-purge47-hardening
branch
from
August 4, 2026 10:54
886522b to
60676c2
Compare
openaxcloud
pushed a commit
that referenced
this pull request
Aug 4, 2026
…ze barrier + topology-drift guard RR-08 refused PR #52 on two paths; both fixed fail-closed with executable tests. 1) Object-storage write barrier was bypassable. POST /projects/:id/thumbnail/ upload-url called ensureBucket + createUploadUrl with NO freeze check, so it (and the background thumbnail capturer, and any future signed-upload route) could recreate a bucket/object AFTER the purge zero-check. Fixed structurally: - guardObjectStorageWrites() wraps the storage so every CREATE/MODIFY primitive (ensureBucket/createUploadUrl/putObject/moveObject) REFUSES a frozen project with OBJECT_STORAGE_PURGE_FROZEN (→ 403). resolveObjectStorage() returns the guarded wrapper for ALL routes + the capturer; the purge's own erasure uses the RAW adapter (resolveRawObjectStorage) so it can still delete the project it froze. - the thumbnail route also calls the explicit objectStorageWriteBlocked guard. Tests: tests/object-storage-purge-freeze.spec.ts (thumbnail→403, upload-url→403, reads still 200, unfreeze→200); object-storage.spec.ts guardObjectStorageWrites unit (writes refused, reads/deletes pass, unfrozen project writes). 2) Topology wasn't serialized against the tombstone. The external GCS/PVC erasure ran on the PRE-transaction sole/shared topology; the tx recomputed it independently, so a membership race (shared→sole / sole→shared) during erasure could strand a newly-sole org's bucket or destroy a newly-shared org's bucket yet still stamp purgedAt. Fixed: resolveStorageTopology() computes a stable fingerprint; the purge tx re-derives it under the advisory lock and THROWS ACCOUNT_PURGE_TOPOLOGY_DRIFT before any delete/tombstone on drift — account stays queued, next run recomputes fresh (idempotent erasure). Never finalize on a stale inventory. Tests (real Postgres): tests/account-purge-db.spec.ts (6) shared→sole and (7) sole→shared — mutate membership inside the eraseStorage hook (the real race window) and assert abort + no tombstone + intact storage rows. Root typecheck + CI-strict build (tsc src/server.ts) clean; 76 affected api tests + 6 real-PG purge tests + 28 object-storage tests green. PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Aug 4, 2026
…revious synchronize) No code change; forces a fresh pull_request synchronize so Production CI / PR Validation / Code Quality / Security / Preview run on the RR-08 fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
pushed a commit
that referenced
this pull request
Aug 4, 2026
…e erasure RR-09 refused #52: the RR-08 drift guard fired AFTER the irreversible GCS/PVC deletion, so in sole→shared the bucket was already destroyed (the guard only blocked the tombstone); and the object-storage freeze was never released, leaving a residual freeze on abort. Reordered + hardened, fail-closed: (1) GUARANTEE BEFORE DELETION — acquirePurgeGuarantee() runs first, in one tx under the per-user advisory lock: computes the authoritative sole/shared topology AND freezes it (membership for every org the subject belongs to + object storage for the sole-org buckets) atomically, records a recoverable plan. The erasure then runs on THIS locked inventory only. (2) MEMBERSHIP BLOCKED DURING ERASURE — addMember/removeMember take the freeze-set advisory lock and refuse (MEMBERSHIP_FROZEN_FOR_PURGE) for a frozen org, so no join/leave can flip sole↔shared mid-erasure. (3) DELETE ONLY AFTER THE GUARANTEE — eraseStorage is invoked only once a guarantee is held, on guarantee.bucketProjectIds. (4) RECOVERABLE STATE MACHINE — releasePurgeGuarantee() runs in a finally on EVERY exit (purged/drift/throw): unfreeze membership + object storage, clear the plan. reconcilePurgeFreezes() (run at purge-executor start) releases a plan left by a crashed run — no residual freeze. Object-storage freeze moved out of createWriteBarrier into the store guarantee; the barrier now only freezes pods. Tests (real Postgres, account-purge-db.spec.ts): (6) shared bucket never handed to the erasure + survives; (7) co-member cannot leave while frozen; (8) new member cannot join while frozen; (9) NO residual freeze after a failed purge + org writable again; (10) reconciler releases a crashed run's freeze. RR-08 drift check kept as a defence-in-depth backstop. Root typecheck (all services) + CI-strict build clean; 85 affected api tests green (incl. 9 real-PG purge). PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
openaxcloud
force-pushed
the
vague3-purge47-hardening
branch
from
August 4, 2026 13:58
60676c2 to
1b887d8
Compare
openaxcloud
pushed a commit
that referenced
this pull request
Aug 5, 2026
…4 (8 corrections) docs/parity/incoming/EVIDENCE_PR52_aebc914c.md — the 8 corrections (P1-P8), migrations 0085 (PurgeReceipt) + 0086 (workspace barrier), full PR diff, full source of prisma-store.ts + account-storage-purge.ts + manager.ts at code head aebc914, coherent GCS + K8s (PV/disk gone) proofs, raw real-PG/module/workspace-manager output of the new deterministic race tests (18,25,26,27,P4x2,P8x3,P3x2), and B.6 CI (all substantive checks green on aebc914). Supersedes the prior EVIDENCE file; regenerates DOCUMENT_MANIFEST.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
…MQ + preuve d'effacement auditée Comble le gap HAUTE de SECURITY_PRIVACY_COMPLIANCE v3 (§Rétention) : la machine request→grâce 14 j→ready_to_purge existait mais RIEN ne consommait ready_to_purge — purgedAt n'était jamais écrit hors des specs. - account-purge.ts (pur) : preuve d'effacement structurée par classe (supprimées/anonymisées/conservées+motif, vérif « 0 ligne restante »). - store.purgeUserAccount (prisma + in-memory) : purge réelle classe par classe dans UNE transaction pg_advisory_xact_lock par user — idempotente, sûre en concurrence (2 workers → 1 purge) ; rétention financière 7 ans fail-closed (canPurgeFinancialRecord) ; audit logs RÉDIGÉS jamais supprimés ; ledger 0078 immuable respecté (retenu + consigné) ; tombstone User anonymisé portant purgedAt ; recomptage post-purge par classe, toute ligne restante ⇒ rollback complet. - POST /internal/account-purge (secret interne, DRY-RUN par défaut, ACCOUNT_PURGE_ENABLED=true pour armer) : consomme la file account.pendingDeletionUserIds, persiste la preuve dans l'AdminAuditLog (account.purge_completed) AVANT de sortir l'id de la file. - worker : job account.purge (enterprise-jobs) + CronJob Helm accountPurge (30 4 * * *) — même patron que inactivity.gc. - tests : 9 route-level négatifs d'abord (fenêtre non échue, annulation, dry-run, double exécution, course, rétention consignée, purge complète, org partagée) + 4 preuves DURABLES vrai Postgres (SQL 0-ligne, preuve relue de la DB, 2 clients en course, trigger ledger refuse DELETE). - preuve PG jouée sur pgvector:pg16 jetable : docs/deploy-evidence/ 2026-07-22-account-purge-worker/ (journal + logs + preuve JSON). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… workspace PVCs) Closes the physical half of the §16.12 deletion gap left by PR #43 (which erases DB rows only). eraseProjectsStorage() erases a purged account's per-project object-storage buckets and workspace volumes with the same evidence discipline as the DB purge — list BEFORE → delete → re-count AFTER — and emits PurgeClassReport{object_storage,workspace_volumes} so it folds straight into ErasureProof.verifiedZeroRemaining. FAIL-CLOSED: any bucket/volume that doesn't re-count to 0 (or whose delete threw) leaves remainingAfterPurge > 0. Idempotent (missing bucket/workspace = verified no-op) so the worker can retry. All I/O behind injected ports → unit-testable + replayable against a throwaway bucket/volume, never prod data. 6 tests: happy-path evidence, fail-closed (bucket won't delete / workspace throws), idempotent no-op, empty-account vacuous verify, multi-project aggregation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Threads an eraseStorage hook through purgeUserAccount (both the Prisma store and the in-memory test store) and the /internal/account-purge route: - BEFORE any DB tombstone is stamped, the account's sole-org projects' GCS buckets + workspace PVCs are erased and re-counted; if ANY remains the purge throws (ACCOUNT_PURGE_PHYSICAL_INCOMPLETE) → the account stays queued and is retried. An account is thus only ever marked purged once BOTH its rows and its physical storage are proven gone. - The object_storage / workspace_volumes evidence classes fold into the existing ErasureProof, so verifiedZeroRemaining now covers physical storage too. - Physical I/O runs outside the Postgres tx (GCS/PVC deletes aren't transactional) and is idempotent, so a retry is safe. - Workspace PVCs are deleted via workspace-manager (the API pod has no k8s access); object storage via the existing ObjectStorage.deleteBucket. - The purger is injectable (ApiAppOptions.accountStoragePurger) so route tests exercise the fail-closed gate + proof embedding without live GCS/manager. Existing purge suite stays green; +2 route tests (fail-closed; proof embeds object_storage/workspace_volumes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…volume) A tamper-evident, reproducible proof of physical erasure — mirrors PR #43's DB evidence discipline for storage. - scripts/physical-purge-proof.ts: runs the REAL eraseProjectsStorage against a throwaway test bucket + workspace volume (never prod data) — seeds objects, lists BEFORE, erases, re-counts AFTER (0), folds into an ErasureProof, and emits canonical proof.json + its SHA-256. Deterministic (fixed ids/timestamps). - docs/deploy-evidence/2026-07-22-account-physical-purge/{README,proof.json, SHA256SUMS}: the committed artifact (5 objects → 0, 1 workspace → 0, verified). JSON, not an ignored .log. - src/tests/physical-purge-proof.spec.ts: CI replays it and asserts the committed artifact reproduces byte-for-byte, so the hashed proof can't drift from code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DB integration test (account-purge-db.spec) exercises the /internal/account- purge route, which now runs the physical-erasure gate. Without an injected purger the route's default eraser fetches a non-existent workspace-manager in CI and (correctly) fails the purge closed → the real-PG test saw purged:0/failed:1. Fix the cause: inject the REAL eraseProjectsStorage over in-memory fakes (seeded with a bucket + workspace per project), so the fail-closed gate is genuinely exercised (list → delete → verify 0) without a live workspace-manager/GCS. The row-level SQL assertions are unchanged; not disabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Answers the four reserves that refused PR #47, with REAL before/after E2E proof on actual GCS + Kubernetes (not the memory adapters that were rejected). 1. Write barrier (#1): eraseSubjectStorage now calls WriteBarrierPort.freeze BEFORE any delete; a freeze failure aborts the erasure (nothing deleted, not verified) so nothing is recreated between erase/verify and the tombstone. Prod path: workspace-manager POST /workspaces/:id/freeze (revoke agent token + stop pod). 2. Real disappearance (#2): verification re-checks the LIVE backend — GCS list, and a new GET /workspaces/:id/pvc-exists that does a real — never the workspace row's DELETED status (a partial k8s delete can leave a PVC). 3. By data subject (#3): the inventory is per-subject — the subject's sole-org buckets AND their per-user workspace in EVERY project they touched (sole-org + collaborator, via ProjectCollaborator), not just one main workspaceId. 4. Real proof (#4): two replayable, hashed E2Es under WIF-proof guardrails (dedicated test resources, no persistent keys, ~$0, full teardown): - GCS: GcsObjectStorage → eraseSubjectStorage against a throwaway bucket in the test project ecode-proof-b906ss; 3 objects → bucket+objects gone. - Kubernetes: a real Bound PVC on a throwaway local kind creates and manages local Kubernetes clusters using Docker container 'nodes' Usage: kind [command] Available Commands: build Build one of [node-image] completion Output shell completion code for the specified shell (bash, zsh or fish) create Creates one of [cluster] delete Deletes one of [cluster] export Exports one of [kubeconfig, logs] get Gets one of [clusters, nodes, kubeconfig] help Help about any command load Loads images into nodes version Prints the kind CLI version Flags: -h, --help help for kind -q, --quiet silence all stderr output -v, --verbosity int32 info log verbosity, higher value produces more output --version version for kind Use "kind [command] --help" for more information about a command. cluster → deleted → verified gone via live (kind, not GKE, so $0 — no cost sign-off needed per the guardrail). Artifacts + SHA256 in docs/deploy-evidence/2026-07-23-physical-purge-e2e/. Removes the old in-memory proof the expert rejected. Module + route + real-Postgres suites green (33 tests). PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… fail-closed + E2E negatives 1. k8s barrier fails on ANY delete failure: freezeWorkspace attempts every revoke but THROWS if any rejected (never claims the barrier / marks the row stopped while a write path may be live). Negative: manager.spec reserve #1. 2. Real GCS backend required: eraseSubjectStorage refuses (unverified, never deletes) when buckets exist but no active backend — a NoopObjectStorage can never certify absence. Negatives: module spec + REAL GCS E2E (inert backend refused, bucket survived). 3. Block ALL object-storage writes during purge: the barrier marks projects purge-frozen; upload-url/ensure-bucket/move return 403 OBJECT_STORAGE_PURGE_ FROZEN, so nothing is recreated after the zero-check. 4. Inventory by REAL authorization: workspaces for EVERY project in ANY org the subject is a member of (shared orgs too, without a ProjectCollaborator row), plus explicit collaborations. 5. Only an authenticated NotFound = absence: pvcExists no longer swallows k8s.get errors — the client returns undefined only on a real NotFound and re-throws network/RBAC errors, so a read error fails closed. Negatives: manager.spec reserve #5 + kind E2E (surviving PVC reported present). 6. E2E negatives: real GCS + kind E2Es each carry a negative; the #1/#5 error negatives are proven deterministically in manager.spec (a k8s error cannot be reliably injected through kubectl/kind). api + workspace-manager typecheck clean; 34 api purge tests + 48 manager tests green; both real E2Es pass with negatives (artifacts hashed in docs/deploy-evidence/2026-07-23-physical-purge-e2e/). PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hiers committés ee0df5a a mis à jour gcs/k8s-proof.json ET leurs SUMS, mais les SUMS committés ne correspondaient pas aux proofs committés (index écrit avant la dernière régénération des proofs). Les proofs sont INCHANGÉS ici — seul l'index de hash est recalculé pour être vérifiable par sha256sum -c. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ze barrier + topology-drift guard RR-08 refused PR #52 on two paths; both fixed fail-closed with executable tests. 1) Object-storage write barrier was bypassable. POST /projects/:id/thumbnail/ upload-url called ensureBucket + createUploadUrl with NO freeze check, so it (and the background thumbnail capturer, and any future signed-upload route) could recreate a bucket/object AFTER the purge zero-check. Fixed structurally: - guardObjectStorageWrites() wraps the storage so every CREATE/MODIFY primitive (ensureBucket/createUploadUrl/putObject/moveObject) REFUSES a frozen project with OBJECT_STORAGE_PURGE_FROZEN (→ 403). resolveObjectStorage() returns the guarded wrapper for ALL routes + the capturer; the purge's own erasure uses the RAW adapter (resolveRawObjectStorage) so it can still delete the project it froze. - the thumbnail route also calls the explicit objectStorageWriteBlocked guard. Tests: tests/object-storage-purge-freeze.spec.ts (thumbnail→403, upload-url→403, reads still 200, unfreeze→200); object-storage.spec.ts guardObjectStorageWrites unit (writes refused, reads/deletes pass, unfrozen project writes). 2) Topology wasn't serialized against the tombstone. The external GCS/PVC erasure ran on the PRE-transaction sole/shared topology; the tx recomputed it independently, so a membership race (shared→sole / sole→shared) during erasure could strand a newly-sole org's bucket or destroy a newly-shared org's bucket yet still stamp purgedAt. Fixed: resolveStorageTopology() computes a stable fingerprint; the purge tx re-derives it under the advisory lock and THROWS ACCOUNT_PURGE_TOPOLOGY_DRIFT before any delete/tombstone on drift — account stays queued, next run recomputes fresh (idempotent erasure). Never finalize on a stale inventory. Tests (real Postgres): tests/account-purge-db.spec.ts (6) shared→sole and (7) sole→shared — mutate membership inside the eraseStorage hook (the real race window) and assert abort + no tombstone + intact storage rows. Root typecheck + CI-strict build (tsc src/server.ts) clean; 76 affected api tests + 6 real-PG purge tests + 28 object-storage tests green. PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…revious synchronize) No code change; forces a fresh pull_request synchronize so Production CI / PR Validation / Code Quality / Security / Preview run on the RR-08 fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e erasure RR-09 refused #52: the RR-08 drift guard fired AFTER the irreversible GCS/PVC deletion, so in sole→shared the bucket was already destroyed (the guard only blocked the tombstone); and the object-storage freeze was never released, leaving a residual freeze on abort. Reordered + hardened, fail-closed: (1) GUARANTEE BEFORE DELETION — acquirePurgeGuarantee() runs first, in one tx under the per-user advisory lock: computes the authoritative sole/shared topology AND freezes it (membership for every org the subject belongs to + object storage for the sole-org buckets) atomically, records a recoverable plan. The erasure then runs on THIS locked inventory only. (2) MEMBERSHIP BLOCKED DURING ERASURE — addMember/removeMember take the freeze-set advisory lock and refuse (MEMBERSHIP_FROZEN_FOR_PURGE) for a frozen org, so no join/leave can flip sole↔shared mid-erasure. (3) DELETE ONLY AFTER THE GUARANTEE — eraseStorage is invoked only once a guarantee is held, on guarantee.bucketProjectIds. (4) RECOVERABLE STATE MACHINE — releasePurgeGuarantee() runs in a finally on EVERY exit (purged/drift/throw): unfreeze membership + object storage, clear the plan. reconcilePurgeFreezes() (run at purge-executor start) releases a plan left by a crashed run — no residual freeze. Object-storage freeze moved out of createWriteBarrier into the store guarantee; the barrier now only freezes pods. Tests (real Postgres, account-purge-db.spec.ts): (6) shared bucket never handed to the erasure + survives; (7) co-member cannot leave while frozen; (8) new member cannot join while frozen; (9) NO residual freeze after a failed purge + org writable again; (10) reconciler releases a crashed run's freeze. RR-08 drift check kept as a defence-in-depth backstop. Root typecheck (all services) + CI-strict build clean; 85 affected api tests green (incl. 9 real-PG purge). PROVEN_REVIEW_PENDING. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
acquirePurgeGuarantee took the account-purge lock, READ topology, then took the membership freeze-set lock. addMember/removeMember sync on the freeze-set lock, not account-purge — a join could grab it first, commit, and the read topology was stale → sole bucket erased before the drift check. Fix: take the membership freeze-set lock BEFORE resolveStorageTopology() and hold to commit (read+freeze atomic). Canonical lock order: account-purge < membership < objectStorage. Test (real PG, (11)): deterministic — purge confirmed blocked on the membership lock (pg_locks NOT granted) before reading topology; a slipped-in join is reflected; bucket EXCLUDED from eraseStorage + survives; no residual freeze. 3/3. Root typecheck + CI-strict build clean; 10 real-PG purge + 34 object-storage green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a plan before both thaws
REVIEW_BLOCKED deep audits found two real holes; both fixed, both re-proven (real PG).
A.1 (global OrganizationMember scan): 3 write sites repo-wide — addMember (upsert) +
removeMember (deleteMany), both already lock system-setting:membership.purgeFrozenOrgIds
and cover ALL invite/import/admin/SCIM/role routes, AND the purge tombstone
organizationMember.deleteMany({ where: { userId } }) in the finalize tx, which did NOT
lock. Reopened race: the tombstone could flip an org's member count DURING another
purge's atomic read→freeze section. FIX: the finalize tx now takes the membership
freeze-set lock right after account-purge (order account-purge < membership <
objectStorage), serialising with every guarantee.
A.2 (partial-thaw recovery): releasePurgeGuarantee/reconcilePurgeFreezes deleted the
plan row UNCONDITIONALLY even when a freeze remove failed (swallowed) — the plan is
the only durable pointer to a frozen id, so that stranded the freeze forever. FIX:
deletePurgePlanIfFullyThawed() deletes the plan ONLY when neither freeze set still
holds any of its ids; else the plan is kept for the reconciler.
Tests (real PG): (12) membership-thaw fails → plan kept + reconciler recovers;
(13) crash between thaws → plan kept + recovers; (14) reconciler never deletes a plan
while a thaw fails. 13 purge tests (incl. (11) deterministic concurrent) + 34
object-storage green; root typecheck + CI-strict build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-10 A.1+A.2) docs/parity/incoming/EVIDENCE_PR52_1b887d88.md — self-contained package the expert can read without repo access: A.1/A.2 audit results, canonical lock order, full PR net patch + the CODEX-10/A.1/A.2 delta, full source of prisma-store.ts and account-purge-db.spec.ts at the code head, raw output of the deterministic concurrent test (11), and the raw real-Postgres purge suite (13, incl. A.2 recovery 12/13/14) + 34 object-storage tests. B.6 carries the CI conclusions for code head 1b887d8. Docs-only; the code is identical to 1b887d8. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-plan safety) Global freeze-sets (id-lists) could not tell whose freeze was whose: two purges sharing an org/project meant releasing plan A lifted a freeze plan B still needed. Replaced with an OWNERSHIP model (migration 0083_purge_plan_ownership): - PurgePlan: one row per active purge, with ownerToken + leaseExpiresAt + version (CAS reclaim). - PurgeFreeze: one row per (resourceType, resourceId, planId) [unique] — each frozen resource OWNED by exactly one plan; a resource is frozen iff >= 1 row. Guarantees by construction: - release deletes ONLY the plan's own freeze rows → a shared org stays frozen while another live plan owns it; addMember/removeMember refuse while >= 1 plan freezes it. - reconciler reclaims ONLY lease-EXPIRED plans, via CAS on version (a live plan — even one blocked in a slow erasure — is never touched; two reconcilers can't double-reclaim); deletes only the reclaimed plan's rows. - object-storage route guard now queries store.isObjectStorageProjectPurgeFrozen (PurgeFreeze count), not a global list. Advisory lock renamed purge:membership-freeze; canonical order account-purge < membership preserved (guarantee + finalize tombstone). Tests (real PG, account-purge-db.spec.ts): (15) two plans share an org → releasing one keeps it frozen until the LAST releases; (16) reconciler never reclaims a live plan; (17) reclaims an abandoned plan via CAS (2 concurrent reconcilers → once), only its own resources; (18) crash between thaws → plan recoverable, idempotent reprise, zero residual, no other plan touched. 14 purge + 58 affected tests green; root typecheck + CI-strict build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…per-plan ownership) docs/parity/incoming/EVIDENCE_PR52_531f2461.md — the ownership model (schema + migration 0083), full PR net patch + the RR-1bd27929 source-only delta, full source of prisma-store.ts + account-purge-db.spec.ts at code head 531f246, raw real-PG output of the multi-plan tests (15-18) + concurrent test 11 + the 14-test purge suite + 34 object-storage tests, and B.6 CI (all substantive checks green on 531f246). Regenerates DOCUMENT_MANIFEST.yaml (fileCount 173). Docs-only; code identical to 531f246. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ verified reconcile Three hardening reserves on the per-plan model (migration 0084_purge_plan_lease_singleton: PurgePlan.status + UNIQUE userId): 1. LIVE LEASE. leaseExpiresAt was set once, never renewed → a slow erasure (>TTL) let a reconciler reclaim a live plan. Now the owner runs a background heartbeat renewing the lease via CAS (renewPurgeLease: id+ownerToken+version+status=ACTIVE) every renewIntervalMs (<< TTL); a failed renewal marks the lease LOST. A guard revalidates ownership+lease BEFORE each irreversible bucket/PVC delete (threaded into eraseSubjectStorage via deps.guard), before the finalize tx, and immediately before the tombstone — after loss it aborts (no delete, no finalization). 2. SAFE RECLAIM + PER-USER SINGLETON. userId is UNIQUE (one plan/subject); acquirePurgeGuarantee refuses a second purge (PURGE_ALREADY_ACTIVE) while a plan is live, else reclaims an abandoned one. reconcile reclaims only plans whose lease expired beyond a grace (clock-lag guard), via durable CAS ACTIVE->RECLAIMING (+version) which also invalidates the old owner's next renewal. 3. VERIFIED RECONCILE. No longer swallows cleanup errors as success — increments reconciled ONLY after the plan AND its freezes are verifiably gone; a cleanup failure leaves the plan RECLAIMING (recoverable), not counted. Tests (real PG, account-purge-db.spec.ts): (19) erase>TTL heartbeat renews, concurrent reconciler reclaims nothing; (20) dead owner, 2 reconcilers → exactly one wins; (21) renewal vs reclaim race → one CAS winner; (22) lease lost mid-erase → no further delete, no tombstone, recoverable; (23) two workers same user → one plan, one physical execution, one tombstone+proof; (24) reconciler cleanup failure → reconciled 0, plan RECLAIMING, second pass finishes. 20 purge + 75 affected tests green; root typecheck + CI-strict build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…live lease + singleton + verified reconcile) docs/parity/incoming/EVIDENCE_PR52_dfbe5c68.md — lease/singleton model + migration 0084, full PR net patch + RR-CODEX-12 source-only delta, full source of prisma-store.ts + account-purge-db.spec.ts at code head dfbe5c6, raw real-PG output of tests 19-24 + the 20-test purge suite + object-storage, and B.6 CI (all substantive checks green on dfbe5c6). Regenerates DOCUMENT_MANIFEST.yaml (173). Docs-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…P7,P8) P2 — reclaim in acquirePurgeGuarantee is now a SINGLE conditional DELETE on (id + version + status + leaseExpiresAt < now()−grace), the cutoff computed with the POSTGRES clock. If the old owner's heartbeat renewed between our read and here, it matches 0 rows and the acquirer STOPS — never delete-by-id a renewed plan and start a 2nd physical erasure. Test (25): heartbeat renews precisely between read and delete (via a test seam) → PURGE_ALREADY_ACTIVE, 0 executions. P4 — the lease guard now runs at the LINEARISATION point, immediately before the irreversible deleteBucket/deleteWorkspace (no network call in between), OUTSIDE the per-resource try so a lost lease PROPAGATES (aborts), never silently skips. Tests: loss between listObjects→deleteBucket and pvcExists→deleteWorkspace → delete never called. P5 — transferProject now takes the membership-freeze lock and refuses a project whose storage is frozen, or whose source/target org membership is frozen (assertProjectNot PurgeFrozen). Test (26): transfer refused while frozen, allowed after unfreeze. P6 — the erasure PROOF is written to a new PurgeReceipt table in the SAME tx as the tombstone (migration 0085); the executor removes a purged user from the queue ONLY once hasPurgeReceipt (else counts missingReceipt and keeps it queued). Test (27). P7 — releasePurgeGuarantee is now the SAME model as reconcile: one atomic conditional DELETE of the plan (freezes cascade via FK), guarded by ownerToken. No multi-step partial thaw. Doc rewritten to describe the atomic-delete model exactly. Test (18) rewritten: a failed release keeps BOTH freezes + plan recoverable. P8 — eraseSubjectStorage fully fail-closed: a non-empty inventory REQUIRES a write barrier (absent barrier ⇒ frozen=false) AND real ports (objectStorage.active, workspaceVolumes present); the proof is the absence of the CONTAINER itself (bucketStillExists) not just its content. Tests (P8 x3). Real-Postgres suite 23 tests + module 11 tests green; api typecheck + CI-strict build clean. Point P3 (workspace-manager durable barrier) + evidence coherence follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ier vs reprovision race The workspace freeze deleted Secret/Pod/Service but left NO durable state, so a concurrent startWorkspace could recreate the PVC/Pod after the zero-check and before the tombstone (a "purged" account with a live PVC). Fix (migration 0086 adds WorkspaceRuntime.purgeFrozen + purgeFenceToken): - freezeWorkspace records a DURABLE barrier (purgeFrozen + fence token) ONLY after every live write path is revoked; it also creates a frozen tombstone row when the runtime does not exist yet (blocks a first-time reprovision). - a single choke-point assertNotPurgeFrozen() gates startWorkspace AND restartWorkspace (which every reprovision funnels through) → WORKSPACE_PURGE_FROZEN. - unfreezeWorkspace releases the barrier, FENCED by the owning plan's token. - the api write barrier passes the fence token (userId — valid since the per-user singleton means one active plan per subject); the freeze route threads it through. Tests (manager.spec, deterministic): a start/restart AFTER freeze (zero-check) and BEFORE the tombstone is REFUSED; a wrong fence token cannot unfreeze, the owner's can; freezing a not-yet-provisioned id blocks a first-time start. api 45 + wsm 57 tests green; both services typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…proves PV/disk gone - The README/GCS-script named the test project `ecode-proof-b906ss` while the artifact (and the script default) used `ecode-wif-proof-834022`. Unified to the project the artifact actually records — `ecode-wif-proof-834022` — everywhere. - The K8s E2E now proves the UNDERLYING PV/DISK is gone, not just the PVC binding: StorageClass reclaimPolicy=Delete, and the script polls until `pv=gone` (FAILs if the PV/disk survives). Re-ran on a real throwaway kind cluster → pvc=NotFound AND pv=gone, verified=true (artifact version 3). The kind teardown is explicitly separate from and NOT relied upon for the proof (the PV is verified gone while the cluster is still up). - Both proof artifacts' SHA256SUMS now use one consistent convention (file bytes) and both verify. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4 (8 corrections) docs/parity/incoming/EVIDENCE_PR52_aebc914c.md — the 8 corrections (P1-P8), migrations 0085 (PurgeReceipt) + 0086 (workspace barrier), full PR diff, full source of prisma-store.ts + account-storage-purge.ts + manager.ts at code head aebc914, coherent GCS + K8s (PV/disk gone) proofs, raw real-PG/module/workspace-manager output of the new deterministic race tests (18,25,26,27,P4x2,P8x3,P3x2), and B.6 CI (all substantive checks green on aebc914). Supersedes the prior EVIDENCE file; regenerates DOCUMENT_MANIFEST.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ P6 receipt-gated queue removal P3 (WorkspaceRuntime durable barrier): - R-P3-01: startWorkspace re-checks the barrier LINEARISED immediately before each irreversible k8s create (PVC/Secret/Pod/Service) + a FINAL post-create check that revokes any recreated object; a purge-frozen hit now PROPAGATES (not masked as FAILED). - R-P3-02: freezeWorkspace no longer swallows persistence failures — the durable barrier is CONFIRMED (re-read purgeFrozen=true) before success; a k8s revoke never compensates. - R-P3-03: unfreeze requires the EXACT owning token — an absent/empty token is refused exactly like a wrong one (token-less caller can never lift a fenced barrier). - R-P3-04: /unfreeze route + releaseWorkspaceBarrier wired into the purge release path (every exit), and reconcileStaleWorkspaceFreezes + /internal route make an orphaned barrier recoverable (never durably frozen with no owner). - R-P3-05: the fence is the PER-ATTEMPT ownerToken (not the stable userId) — a delayed release from a prior attempt can't lift a newer attempt's barrier (ABA). P6 (queue removal strictly receipt-conditioned): - All queue removal on a purge/already-purged claim goes through the single primitive removePendingOnlyWithReceipt(userId): removes ONLY if hasPurgeReceipt, else increments missingReceipt and KEEPS the id queued. Used for the 3 paths (pre-state purged, outcome=purged, outcome=already_purged). - TestApiStore mirrors the real store's SAME-TX receipt write. Tests (proven real): wsm 85 pass (+4 P3: orchestrated race, token-omitted, ABA, abandon/reconcile). api 1508 pass incl account-purge-db 24 (T27 rewritten through the REAL /internal/account-purge route: missingReceipt=1 + no remove + id still queued; T27b negative: PurgeReceipt.upsert failure rolls back the whole tombstone tx). Migration 0087. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…5 (P3 linearisation + P6 receipt-gated) Regenerated EVIDENCE_PR52_<head>.md for the v5 head: the 2 fixed reserves (P3 R-P3-01..05 barrier linearisation, P6 removePendingOnlyWithReceipt), all three migrations (0085/0086/0087), full source of the key files, the rewritten T27/T27b, and the raw real-Postgres + workspace-manager test output (wsm 85, api 1508, account-purge-db 24). Manifest regenerated; stale v4 evidence removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openaxcloud
force-pushed
the
vague3-purge47-hardening
branch
from
August 5, 2026 16:30
9c91cd1 to
3bd148b
Compare
Contributor
…ures fail-closed (R-P3-06) Réserve BLOQUANTE de l'expert sur la PR #52 (SHA audité 3bd148b). `unfreezeWorkspace` LISAIT le fence token puis émettait un `UPDATE ... WHERE id = ?` INCONDITIONNEL. Une tentative de purge N0 retardée entre ces deux instructions écrasait la barrière que N1 avait installée entre-temps : attendu purgeFrozen=true / token=owner-N1, obtenu purgeFrozen=false — barrière tombée en pleine fenêtre d'effacement (ABA). Le reconciler « stale » portait le même TOCTOU sur son snapshot, et plusieurs lectures de barrière faisaient `.catch(() => undefined)` : une erreur DB était indiscernable de « pas de barrière » → elles échouaient OUVERTES. Pourquoi le test R-P3-05 existant ne l'attrapait pas : il jouait l'unfreeze retardé SÉQUENTIELLEMENT, donc la lecture voyait déjà le nouveau token et le contrôle applicatif refusait. Le défaut n'apparaît que si le nouveau freeze tombe DANS la fenêtre lecture→écriture. 1. CAS atomique — `WorkspaceStore.releasePurgeFence(id, token)` : un seul UPDATE conditionnel `WHERE id AND purgeFrozen AND purgeFenceToken` ; 0 ligne affectée ⇒ on ne dégèle pas. Suit le précédent `claimMeterWindow` du même store. 2. CAS équivalent pour le reconciler — `releaseStalePurgeFence(id, {fenceToken, frozenAt})` : CAS sur la version exacte du snapshot ayant servi au verdict de staleness ; un re-freeze après le scan change token ET frozenAt → 0 ligne. 3. Lectures fail-CLOSED — `assertNotPurgeFrozen` lève WORKSPACE_PURGE_BARRIER_UNVERIFIABLE, `isPurgeFrozen` renvoie true, et la lecture de `freezeWorkspace` refuse. Plus aucune n'avale une erreur DB. 4. Tests d'interleaving sur PostgreSQL 16 RÉEL — `purge-fence-cas.integration.spec.ts` (14 tests, 2 clients Prisma/PG distincts = 2 backends) + 4 tests unitaires jouables en CI sans DB. Bug de bord attrapé DANS ce correctif : déplacer le test de propriété dans le WHERE change le sens du jeton (opérande SQL, plus valeur lue en JS). Deux tests de truthiness préexistants repliaient un jeton VIDE sur « pas de jeton » — `rowToRecord` et le reconciler. Combinés, une barrière fencée par '' aurait comparé à NULL, matché 0 ligne et serait devenue DÉFINITIVEMENT irrécupérable (pire qu'avant, où l'update inconditionnel la reprenait toujours) ; atteignable en principe : app.ts gèle avec `fenceToken ?? ''`. Corrigé en `!== null` / `!== undefined` + test dédié — c'est ce test, écrit d'abord, qui a fait tomber la version intermédiaire du correctif. Chaque test joue la version LEGACY et la version CAS sous le MÊME interleaving, injecté au MÊME point : les assertions legacy prouvent que le banc déclenche vraiment la course. Les 6 tests neufs ont été rejoués contre le code d'avant correctif → tous en échec (dont « le start résout avec succès » : Pod vivant sur un runtime dont la barrière est illisible = fail-open démontré). Préservés : R-P3-03 (un appelant sans token ne lève jamais une barrière fencée) et R-P3-04 (une barrière réellement orpheline reste réclamable), tests dédiés. SQL littéral capturé (log_statement=all) dans le paquet de preuves. Piège traité + testé : Prisma émet `purgeFenceToken IS NULL` (pas `= NULL`) pour le token nul — sinon une barrière sans token serait INDÉGELABLE jusqu'au reconciler 24h. Suite complète workspace-manager (vraie DB attachée) : 110/110, exit 0. Typecheck strict OK. Preuves rejouables : docs/deploy-evidence/2026-08-07-purge-fence-cas/ Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hine (R-P3-06) Le client « mort » du bloc fail-closed pointait sur un hôte/port codés en dur (127.0.0.1:32769 — le conteneur PG d'une autre session, désormais disparu). Le test passait quand même, mais pour la mauvaise raison (port fermé) et il n'était pas rejouable ailleurs : sur une autre machine il aurait échoué sur un refus de connexion, pas sur l'erreur serveur qu'on veut prouver. Désormais dérivé de DATABASE_URL : mêmes hôte/port/identifiants que la base vivante, seul le nom de base est remplacé par un nom impossible. L'erreur est donc une VRAIE erreur serveur sur un serveur JOIGNABLE (« Database does_not_exist_purge52_r_p3_06 does not exist on the database server »), et la preuve se rejoue telle quelle avec n'importe quel DATABASE_URL. Sortie brute régénérée. 14/14 sur le spec d'interleaving, 110/110 sur le service. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the four expert reserves that refused PR #47 (physical account purge) (§D of the V3 review), with REAL before/after E2E proof on actual GCS and Kubernetes — replacing the memory-adapter proof that was rejected. Rebased on
main. PROVEN_REVIEW_PENDING — no merge without green light.eraseSubjectStoragecallsWriteBarrierPort.freezebefore any delete; a freeze failure aborts erasure (nothing deleted, not verified), so nothing can be recreated between erase/verify and the tombstone. Prod path: workspace-managerPOST /workspaces/:id/freeze(revoke agent token + stop pod).list, and a newGET /workspaces/:id/pvc-existsdoing a realget pvc— never the workspace row'sDELETEDstatus (a partial k8s delete can leave a PVC behind a "deleted" row).ProjectCollaborator), not just one mainworkspaceId.docs/deploy-evidence/2026-07-23-physical-purge-e2e/:GcsObjectStorage→eraseSubjectStorageagainst a throwaway bucket in the test projectecode-proof-b906ss(never prod). 3 objects → bucket+objects gone,verified:true.kindcluster → deleted → verified gone via liveget pvc.Cost & teardown
finallyforce-delete; 0 buckets left).kind(a real k8s API), not GKE, so no heavy/costly cluster and no cost sign-off was needed per the guardrail. Torn down viakind delete clusterEXIT trap.Checks
services/api/scripts/physical-purge-{gcs-e2e.ts,k8s-e2e.sh}).main; the only expected red is the repo-wideProduction E2E(Playwright) breakage, unrelated to purge.Removes the old in-memory
physical-purge-proof(the rejected memory-adapter proof).🤖 Generated with Claude Code