fix(platform): restore three dropped retention behaviours - #3143
Conversation
The Postgres port left three data-retention behaviours behind. An administrator could configure a window and nothing enforced it. Sweep external conversations. The `externalConversations` category was validated and exposed but no sweep ever read `app.conversations`, so inbound and outbound email bodies were kept forever. The new phase-2 sweep ages conversations by `last_message_at_ms`, honours the org legal hold and the two-pass deletion grace, and stays inside `BATCH_LIMIT`. Message rows ride the parent's `ON DELETE CASCADE`. Stored mail attachments do not, and no other sweep can reach them — the mail lane stamps `file_metadata.source` with the connector slug — so the sweep removes them too, leaving files promoted into a Document to the `documents` category. Sweep two-factor attempts. `app.two_factor_attempts` is cleared on a successful verify and on member removal, so a user who failed 2FA and never came back left a permanently stuck row. It now ages out on the same 30-day window as login attempts. `app.two_factor_grace` is left alone on purpose: an absent row mints a fresh grace window, so ageing it out would reward staying away. Restore the 30-day window for login block counters. The port used 90 days for counters and 30 for attempts, holding `email` and `last_ip` three times longer than the attempts they summarise. Nothing needed the longer window: the admin view reads the most recent 200 counters with no time bound, and the lockout lives on `login_attempts.locked_until`. `integration-check.ts` asserts all three against real Postgres, plus the two-pass grace path.
aaa0ca0 to
c333bd0
Compare
|
Rebased onto current
That changed one behaviour worth calling out. A conversation whose attachment could not be released is now held back too — deleting the parent would orphan the file row behind a dangling pointer, which is the exact failure this cascade was added to prevent. A test-isolation bug of mine, caught by comparing against a baseline. The run showed a After the fix, on a real Postgres and MinIO:
Same six failures on both — the warm-MinIO bucket collision, two All three assertions pass: |
An administrator could set an
externalConversationsretention window and nothing ever deleted. Email bodies were kept forever. Two smaller windows were also wrong.Refs #3142.
Why
The category is exposed and validated in three places —
domains/retention/routes.ts:206,domains/governance/settings-tail.ts:317,lib/shared/schemas/retention.ts:20— anddomains/retention/service.tsnever readapp.conversations.app.conversation_messages.contentis NOT NULL text holding inbound and outbound email bodies, plus ametadatajsonb of envelope detail, so this is the heaviest correspondent-PII surface retention governs.Two windows were also off, neither with a note:
login_block_countersaged at 90 days where 0.4 used one 30-day cutoff for attempts and counters together. Those rows holdemailandlast_ip.two_factor_attemptslost its sweep entirely. 0.4 aged it alongside login attempts, added deliberately because "users who failed 2FA and never came back left a permanently-stuck row".What changed
sweepExternalConversationsages byapp.conversations.last_message_at_ms, served by the existingconversations_org_last_messageindex. Same shape as the sweeps beside it:BATCH_LIMITbound, the org-hold skip, the two-pass grace throughlifecycle_status/status_changed_at_ms, wired intoPhase2StatsandsweepOrgPhase2.A conversation that never received a message has no timestamp to age against and is deliberately not a candidate.
jobs/task-list.tsnow uses one 30-day cutoff for all three auth tables.Beyond the brief, because leaving it would have made the sweep worse than not having it: deleting a conversation would have orphaned its mail attachments permanently.
file_metadata.conversation_idhas no foreign key, and the mail lane stampssourcewith the connector slug, sosweepTempFiles— which takes onlysourceuser/agent— can never collect them. The window would have destroyed the email body and left the attachment bytes live behind a dangling pointer. The index and the blob helper both already existed. A file promoted into a Document is left alone, under the samedocument_id IS NULLguardsweepTempFilesuses.Decisions
two_factor_gracegets no sweep. An absent row reads as "no anchor yet", andevaluateTwoFactorEnforcementthen mints a freshnow + gracePeriodDays— so ageing it out would hand a user who already burned their grace a new window for staying away. It carries no PII beyonduser_id, and member removal already deletes it. A mutation locks this in, so nobody adds the sweep later by symmetry.30 days for counters, not 90.
listBlockCountershas no time window, so nothing reads a counter beyond 30 days, and the lockout itself lives onlogin_attempts.locked_untilrather than a counter bucket — so the shorter window releases no lockout early.Risk
The retention sweeps delete blobs best-effort, so a failed blob delete is logged and the row is destroyed anyway: the file row goes while the bytes may remain. I kept that idiom to match the neighbouring sweeps rather than diverge inside one file, but it is knowingly weaker than the strict posture #3135 gives erasure. Observed directly — with no object store configured the sweep warns and continues. Worth a deliberate decision rather than inheriting it.
Tests
Three assertions in
checkRetention, and eleven mutations, each named with the assertion that went red:frozenConv=0 (want 1)created_at_msON DELETE CASCADEattachment=1 (want 0)two_factor_graceage delete addedgrace=0 (want 1)firstPass=unmarkedfirstPass=ROW GONE— deleted inside the windowThe grace path had no coverage before this — the existing
checkRetentionrunsdeletionGraceDays: 0throughout, sosweepContacts's grace path is untested too. Rather than ship an untested path, there is now an assertion for it.Proven with a driver importing the real
runRetentionCleanupand the real task-list handler against a throwaway database, because re-running the full suite eleven times was not viable. Throwaway role and databases dropped afterwards.Scope
No per-row audit rows. 0.4's
deleteExpiredExternalConversationemittedexternal_conversation.retention_deleted, and no 0.5 sweep emits per-row audit events, so this matches its neighbours instead of introducing a lone pattern. Retention emitting no audit rows at all is tracked in #3142.chatFilterEventsis the same defect shape and is left alone: it has no production writer yet, so it should land with one.Gate:
typecheck,oxlint --type-aware,oxfmt --checkgreen.