Scaling improvements to allow very large collection pushes and queries

[isTravis]

A large test corpus, ~3M records, 5.5 GB could not complete: four separate memory ceilings sat below it and the manifest arrived as a single ~360 MB JSON body.

Nothing in the push protocol changes for existing clients. The CLI, mirror-sync, pushVersion are untouched and take the same paths they always did. Everything new is opt-in.

Read paths

• record_id and type are denormalized onto version_records, indexed (version_id, record_id) and (version_id, type, record_id). Record listing, ?type= filtering and the diff/delta set operations no longer join through record_objects just to order or filter. The added/removed queries now plan as a merge anti-join between two index-only scans where they were correlated NOT EXISTS lookups.
• ?since= deltas and /diff are keyset-paginated. A delta of any size can be walked to completion. truncated is retained and now mirrors pagination.hasMore, so clients that treat a capped delta as "rebuild from scratch" keep working unchanged.
• typeCounts is stored on the version row at commit. Fixes pagination.total under ?type= (it reported the whole-version count — 112,747 where the real answer was 25,341) and removes the per-page-view COUNT(*) GROUP BY on the collection page.
• Enumeration caps raised: records 1,000 → 2,000/page, manifest 50,000 → 100,000. Walking a collection is bounded by request count, not bytes.
• Statement timeouts on every version route surface as 503 + Retry-After rather than a 500.
• Fork copies its record set with one INSERT … SELECT instead of round-tripping every row through the app.

Push path

• Commit streams. It walks the manifest in keyset batches, writes each record's post-strip hash and public content-address back onto its session-manifest row, folds both version digests over hashes streamed from Postgres in sorted order, and populates version_records server-side. Commit memory is now flat in collection size — measured at ~745 MB throughout a 500k commit, where it previously climbed for the duration.
• Commit skips work it already did. Records validated on arrival through /records are not re-validated, and records inherited from the base version are only re-validated when the schema set changed.
• Async finalize. POST …/commit?async=true returns 202 and builds the version in the background; poll GET …/negotiate/:sessionId until status is committed (with result) or failed (with error). The version is invisible to readers until the finalize completes, and a finalize whose process dies is swept by tool:cleanupSessions.
• Chunked manifest upload. Declare manifest_expected instead of sending manifest, then POST JSONL chunks to …/negotiate/:sessionId/manifest. Each response reports which records from that chunk are needed, so bodies can start flowing before the manifest is complete. Chunks are idempotent by hash. Inline manifests are capped at 500,000 entries (413 above that).
• Session expiry is now an idle timeout — every manifest chunk and record batch pushes it back. It was a wall-clock cap that a multi-million-record push would have blown through halfway.

Correctness

The version hash is the protocol's content-address, shared with the CLI, so a streaming implementation that merely looks right is worthless. Pushing byte-identical content through the old and new commit paths produces byte-identical hash and public_hash at 2k, 3k, 500k and 1M records, including:

• the privacy fixture (private type, private field, record-level private record) — identical public_hash and identical public_record_hash population;
• commit-time strip_unknown_fields, where records are rehashed mid-walk — a narrowed-schema stripping push lands on exactly the hash of a clean push of already-narrow records;
• a chunked push vs the inline push of the same content;
• an async commit vs a synchronous one.

VersionHashStream is additionally locked to computeVersionHash by unit tests covering the golden value, empty sets, duplicate hashes, JSON-escaping edge cases and 200 randomized rounds. 97 tests pass.

Measured

	Before	After
Commit memory, 500k	climbs 1.12 → 1.32 GB	flat ~745 MB
Commit wall-clock, 500k	58 s	59–66 s
Negotiate peak, 500k	798 MB attributable	469 MB, and flat in N
Keyset page latency at 3.1M depth	—	~0.7 s for 1,000 records
Largest push completed	~100k	3,113,504

Chunked negotiate peak is flat rather than linear: 303 MB at 100k, 469 MB at 500k, 381 MB at 1M.

Also

• Postgres gets 1 GB of /dev/shm in both compose files. Docker's 64 MB default is too small for parallel query workers once tables are large — a parallel aggregate over 500k rows failed with could not resize shared memory segment. This would have bitten the first large query in production regardless of this branch.
• App heap 448 MB → 2 GB, container 640 MB → 2.5 GB, both env-overridable (APP_HEAP_MB, APP_MEMORY_LIMIT). Sized for a shared box: dev and prod are co-resident, so the defaults here are paid four times over.
• Docs updated across llms.txt, /protocol, the versions API reference, integration docs, the per-collection agent instructions, README and the OpenAPI descriptions.

Not in this PR

Record bodies still live in Postgres (feature-scaling Phase 1). version_records is now index-dominated at ~395 bytes of index per row, which makes bytea(32) hashes the highest-value remaining storage change. mirror-sync still single-shots the manifest and silently truncates above 10,000 records — pre-existing, and now a ~10-line fix since the response carries pagination.