Skip to content

test(platform): make a truncated itest run fail loudly - #3198

Merged
larryro merged 1 commit into
mainfrom
fix/itest-harness-integrity
Sep 4, 2026
Merged

test(platform): make a truncated itest run fail loudly#3198
larryro merged 1 commit into
mainfrom
fix/itest-harness-integrity

Conversation

@larryro

@larryro larryro commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

The truncation defect

backend:integration could run fewer checks than it contains without saying so. The two-factor lifecycle probe (#3187) drove /two-factor/enable/disable on the suite's shared user. Better Auth 1.6.23 rotates the calling session on disable (and on enable under skipVerificationOnEnable): it issues a new token and deletes the old one, so the harness's fixed cookie died at that step and every later lane 401'd.

Before (origin/main 1ea4cc9e9, fresh tale-db + MinIO): 178 checks recorded (171 PASS / 7 FAIL — six of them dead-session garbage right after the 2FA lane), then

[itest] fatal: SyntaxError: Unexpected non-whitespace character after JSON at position 4
    at async checkAutomations (integration-check.ts:7193)

No tally line. The 76 lanes from checkAutomations onward — checkConnectorOauth, checkConversations, checkOutboundSendLane, governance, REST, SSO/SCIM, the agent lanes, the login-throttle/audit chain, the dev seeder — never ran (~207 checks), and nothing in the output said so.

After (this branch, fresh containers): the run completes end to end — 385/385 checks passed across 119/119 lanes, exit 0. checkConnectorOauth ran (5 PASS lines that were hidden before).

Two changes, so it cannot happen silently again:

  • The 2FA probe acts as a dedicated member. signUpOrgMember (hoisted from checkBlobRefAuthority's local copy) creates the enrollee; the lifecycle calls follow the session rotation like a browser (mergeCookieHeader). The shared session is never touched.
  • runLanes — the truncation guard. Every session-bearing lane runs through it (the lane table is the old sequence 1:1, verified by diffing the extracted names). A lane that throws is recorded as a FAIL naming the lane and how many never ran; after every lane the shared session is asserted alive via /api/auth/get-session. The tally prints RUN TRUNCATED at lane N of M — the tally below covers only those, and the exit code is red. A throw before the lanes is recorded the same way, so the tally always prints.

Probe vs gate: the gate is right, the probes were driving the wrong actor

#3187 put viewerCanWrite (editor-or-above; the 0.4 conversations/conversationMessages RLS write rule) on the first line of every conversation write door, ahead of ownership and visibility. That order is intended: a read-only member may not compose, reply, undo, retry or discard mail under the org's name at all, and #3187's unit tests pin it — member → 403 FORBIDDEN before anything loads; the opaque 404 for a hidden conversation is asserted for a role that passes the gate. Option (b) does not apply: the gate never refuses an authorized composer attaching their own upload — an editor reaches attachment_not_owned untouched.

What was wrong is that two integration probes drove the deeper check with a member, so the gate shadowed what they meant to prove. Silencing them would have deleted the ownership/visibility coverage, so both now assert both layers:

  • checkBlobRefAuthority step 10 (fix(platform): blob refs grant nothing; upload and import lanes gated #3160): member → 403 FORBIDDEN (the gate); editor with the owner's blob → 403 attachment_not_owned (ownership, reached through the gate).
  • checkOutboundSendLane (hidden behind the truncation until now): member undo/retry/discard → 403 (the gate); an editor who cannot open the unassigned conversation → opaque 404 (visibility); rows untouched either way.

Doc drift

environment-reference.md (en/de/fr) still called INSTANCE_SECRET the seed of the "deploy admin key" — the Convex-era role that left with the 0.5 backend — and said deploy fails without it, although tale init/tale deploy generate a missing one. It now describes the 0.5 role: the instance's root secret from which the WebDAV app-password HMAC key is derived at boot unless set explicitly (docker-entrypoint.sh, lib/webdav/hmac-key.ts, #3132), with the sandbox blob-staging tokens signed by a subkey of that derivation; rotating it invalidates every app-password. The wider table re-alignment in those files is the edit hook's oxfmt output.

Gates observed

  • bunx tsc --noEmit (platform): clean.
  • bunx oxlint --type-aware backend/integration-check.ts: clean.
  • backend:integration, fresh tale-db + MinIO, ITEST_S3_ENDPOINT + SANDBOX_LLM_GATEWAY_ADMIN_PASSWORD set: base 178 recorded then fatal; branch 385/385 across 119/119 lanes, exit 0.
  • bun run --filter @tale/docs test: 194/194.

Cross-class discoveries (not touched here)

  • services/platform/env.sh still defines ensure_hex_instance_secret with a Convex-era comment ("generate-admin-key"); nothing calls it, so the 64-hex shape of INSTANCE_SECRET is only what tale init generates, not something boot enforces.
  • format:check does not cover docs/**/*.md, while the edit hook formats them — hence tables on main that the hook re-aligns on first touch.
  • The harness carries ~20 more local sign-up/member-insert copies; only the two lanes this PR touched were moved onto signUpOrgMember.

Squash of the wt-3 batch for rebase onto main; the PR body carries the record.
@larryro
larryro force-pushed the fix/itest-harness-integrity branch from 2f4ce92 to 2a67766 Compare September 4, 2026 05:38
@larryro
larryro marked this pull request as ready for review September 4, 2026 05:38
@larryro
larryro merged commit dac9e71 into main Sep 4, 2026
23 checks passed
@larryro
larryro deleted the fix/itest-harness-integrity branch September 4, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant