fix(database): make CLI db-init deterministically reliable (eliminate scaffold.runtime flake)

[rickylabs]

Summary

• Hardens @netscript/database Prisma migration retry behavior for CLI db init.
• Retries only evidenced transient Prisma schema-engine lifecycle signatures, with capped exponential backoff and a bounded non-interactive child timeout.
• Keeps real schema/database errors non-retriable, including standalone Schema engine exited. failures.
• Records harness research, PLAN-EVAL, IMPL-EVAL, and proof artifacts under .llm/tmp/run/fix-db-init-reliability/.

Root Cause

The residual scaffold.runtime db-init flake was still in the Prisma schema-engine lifecycle path, not Postgres readiness. The db-init logs show Aspire waiting for Postgres and the generated database resource to be running, healthy, and ready before invoking Prisma. The evidenced transient signatures were:

• ERR_STREAM_PREMATURE_CLOSE
• Premature close
• Schema engine exited when coupled with schema-engine-windows.exe cli can-connect-to-database
• a local residual hang where Aspire waited 309069ms for prisma-init-postgres, motivating a bounded Prisma-child timeout so retry can recover before the outer operation timeout

Validation

• deno test --allow-all packages/database/tests/migrate-retry_test.ts
• deno test --allow-all --repeats=100 --fail-fast packages/database/tests/migrate-retry_test.ts
• deno check --unstable-kv packages/database/scripts/mod.ts packages/database/tests/migrate-retry_test.ts
• deno run --allow-read --allow-run .llm/tools/run-deno-check.ts --root packages/database --ext ts,tsx
• deno run --allow-read --allow-run .llm/tools/run-deno-lint.ts --root packages/database --ext ts,tsx
• deno run --allow-read --allow-run .llm/tools/run-deno-fmt.ts --root packages/database --ext ts,tsx
• deno doc packages/database/scripts/mod.ts
• deno doc --lint packages/database/scripts/mod.ts
• deno publish --dry-run --allow-dirty from packages/database
• Five consecutive full rtk proxy deno task e2e:cli run scaffold.runtime --cleanup --format pretty runs passed with Summary: passed=47 failed=0; database.init durations were 30857ms, 33324ms, 32995ms, 36449ms, and 33083ms.

Notes

deno task arch:check still fails before doctrine fitness on pre-existing dependency-centralization drift for divergent @netscript/aspire and @netscript/plugin ranges. This PR does not edit dependency declarations.

[rickylabs]

IMPL — db-init reliability

Root cause

The original transient signature remains Prisma schema-engine lifecycle instability: schema-engine-windows.exe cli can-connect-to-database with ERR_STREAM_PREMATURE_CLOSE / Premature close / Schema engine exited, after Aspire has already waited for the Postgres/database resources.

The residual failure mode that made PR #98 insufficient was that the Prisma child could fail to return control to the retry wrapper at all. Locally this surfaced in scaffold.runtime as:

Error: Timed out waiting for Aspire resource prisma-init-postgres to complete.

That run timed out at database.init after 309069ms, so the old exit-code-only retry never got a chance to classify/retry. Reproduction logs also showed concurrent Aspire dashboard-port noise on another scratch AppHost, recorded as drift; the product fix stays scoped to the db-init Prisma engine lifecycle path.

Fix

• Extended packages/database/scripts/migrate.ts to classify all evidenced transient signatures, including the schema-engine can-connect-to-database command and a bounded timeout diagnostic owned by the migration runner.
• Added a non-interactive per-attempt Prisma child timeout: hung Prisma children are killed after 45000ms and retried through the same signature-scoped policy.
• Switched retry timing to capped exponential backoff with a 5-attempt default budget, keeping worst-case retry time under the outer db-operation timeout.
• Preserved single-shot interactive Prisma behavior and no test-layer retries were added.
• Added focused tests for the widened classifier, timeout option propagation, capped backoff, exhaustion diagnostics, and no-retry real schema errors.

Evidence

Focused/package gates passed:

• deno test --allow-all packages/database/tests/migrate-retry_test.ts
• deno test --allow-all --repeats=100 --fail-fast packages/database/tests/migrate-retry_test.ts
• deno run --allow-read --allow-run .llm/tools/run-deno-check.ts --root packages/database --ext ts,tsx
• deno run --allow-read --allow-run .llm/tools/run-deno-lint.ts --root packages/database --ext ts,tsx
• deno run --allow-read --allow-run .llm/tools/run-deno-fmt.ts --root packages/database --ext ts,tsx
• deno doc packages/database/scripts/mod.ts
• deno doc --lint packages/database/scripts/mod.ts
• deno publish --dry-run --allow-dirty from packages/database

Full runtime proof passed 5 consecutive runs of:

rtk proxy deno task e2e:cli run scaffold.runtime --cleanup --format pretty

Results:

• Proof 1: Summary: passed=47 failed=0; database.init 30857ms
• Proof 2: Summary: passed=47 failed=0; database.init 33324ms
• Proof 3: Summary: passed=47 failed=0; database.init 32995ms
• Proof 4: Summary: passed=47 failed=0; database.init 36449ms
• Proof 5: Summary: passed=47 failed=0; database.init 33083ms

[rickylabs]

IMPL summary — db-init reliability

Root cause:

• Residual failures are still in the Prisma schema-engine lifecycle path, not Postgres readiness. database.init logs show Aspire waiting for the Postgres/database resources to be running, healthy, and ready before Prisma starts.
• Captured/evidenced transient signatures: ERR_STREAM_PREMATURE_CLOSE, Premature close, Schema engine exited coupled with schema-engine-windows.exe cli can-connect-to-database, and a local residual non-terminal prisma-init-postgres hang that hit the outer Aspire wait timeout after 309069ms.
• The previous retry was incomplete: it did not cover the full evidenced schema-engine command signature, had a smaller/linear retry budget, and had no per-attempt child timeout to break a hung Prisma child before the outer operation timeout.

Fix:

• Hardened packages/database/scripts/migrate.ts with a 5-attempt bounded retry, capped exponential backoff, and per-attempt diagnostics/exhaustion logging.
• Added a 45000ms non-interactive Prisma child timeout that kills the child and emits an owned retriable timeout diagnostic so db-init retries a fresh engine process.
• Narrowed retry classification to evidenced transient signatures only: explicit premature-close markers, Schema engine exited only when coupled with schema-engine ... cli can-connect-to-database, and the owned timeout diagnostic. Standalone Schema engine exited. is explicitly non-retriable.
• Interactive Prisma runs remain single-shot and are never retried.

Consecutive-run evidence:

• Five consecutive full rtk proxy deno task e2e:cli run scaffold.runtime --cleanup --format pretty runs passed with Summary: passed=47 failed=0.
• database.init durations across those five runs: 30857ms, 33324ms, 32995ms, 36449ms, 33083ms.
• Additional focused proof: five generated-project db init runs passed, and deno test --allow-all --repeats=100 --fail-fast packages/database/tests/migrate-retry_test.ts passed with no flaky marker.

Validation:

• Scoped database check/lint/fmt wrappers passed.
• deno check --unstable-kv packages/database/scripts/mod.ts packages/database/tests/migrate-retry_test.ts passed.
• deno doc packages/database/scripts/mod.ts, deno doc --lint packages/database/scripts/mod.ts, and deno publish --dry-run --allow-dirty from packages/database passed.
• Separate IMPL-EVAL session wrote impl-eval.md with final verdict PASS.

Caveat:

• deno task arch:check still fails before doctrine fitness on pre-existing dependency-centralization drift for divergent @netscript/aspire and @netscript/plugin ranges. This PR does not edit dependency declarations.

[rickylabs]

Superseded by #152. This PR was opened from the same branch as #145 three minutes after #145 squash-merged, leaving it permanently CONFLICTING. Its only genuine net-new value over main — the retriable-failure classifier refinement in migrate.ts (standalone "Schema engine exited." is now non-retriable) plus its unit test — has been extracted cleanly onto a fresh branch off alpha.10 in #152 (deno.json version drift excluded; all package gates green).