feat(statistics): consume projection statistics in the bundle pipeline

[jcoludar]

Summary

The protspace_web half of the projection-statistics MVP (issue #219). The engine
(protspace) computes per-projection cluster-validity + faithfulness statistics and bakes them into
the .parquetbundle as an optional fifth part; this PR makes the web side produce and tolerate
that part. Rendering of the statistics is a deliberate follow-up — out of scope here.

Engine PR (lands first): #61. This PR depends on it and on a
stats-bearing protspace release. It is safe to merge ahead of that release: the prep step
feature-probes the protspace stats subcommand and no-ops if absent — statistics simply don't
appear yet, and nothing else changes.

What's in this PR

• Bundle reader (@protspace/core data-loader + @protspace/utils): accept 3–5 parts.
  The reader now branches on an empty settings slot (statistics-without-settings ⇒ a zero-byte
  4th part) rather than the raw delimiter count, so the optional fifth statistics.parquet is read
  without error. The statistics part is parsed-but-ignored for now; createParquetBundle
  re-export still drops it (documented).
• Prep service (services/protspace-prep): a best-effort stats step that runs after the
  core bundle is produced and outside the pipeline timeout budget, so it can never cost the job
  or lose the bundle. It has:
  • its own nested timeout (stats_timeout_seconds), caught locally so it never reaches the
    parent handler;
  • a bounded, non-latching version probe for protspace stats (single-flight lock + hard
    timeout + kill of a hung subprocess; transient spawn errors are not latched, so a later job
    retries);
  • an atomic re-bundle — writes a sibling temp .parquetbundle and os.replaces it, so a
    stats timeout/kill mid-write can't corrupt the already-shipped bundle.
• Frontend: a new computing_statistics SSE stage wired through FastaPrepStage and the explore
  runtime (progress creep stopped, 95%, "Computing statistics…").
• Tests: the stale "five-part bundles are rejected" assertion is inverted to accept them;
  added 5-part round-trip + zero-byte settings-slot coverage; services/protspace-prep/tests
  exercise the success / timeout / probe-absent paths.
• OpenSpec: openspec/changes/add-projection-statistics/ (proposal, design, spec, tasks).

Verification

• @protspace/core bundle suite: 24 passed (incl. the inverted 5-part + zero-byte-slot tests).
• protspace-prep pipeline suite: 19 passed.
• Pre-commit (lint-staged && quality && docs:annotations:check && docs:build): green.

Data-format change

Additive, backward compatible — existing 3- and 4-part bundles read and write unchanged.

Refs #219

[peymanvahidi]

Drive-by from #329 (docs consolidation) — your part-ordering assumption is now verifiable in-repo, and it holds.

bundle.ts in this PR carries:

ASSUMPTION (unverified in this repo): part-index 3 = settings and part-index 4 = statistics […] That CLI lives outside this repo (services/protspace-prep shells out to protspace bundle / protspace stats), so this ordering has not been verified against the pinned protspace version.

Since the monorepo merge the writer is in-tree at apps/protspace/src/protspace/data/io/bundle.py, and it confirms all three parts of the assumption:

# module docstring, L8-11
# Positional layout: ``core(3) + settings? + statistics?``.  When statistics are
# present but settings are absent, the fourth part is written as **zero bytes** so
# the statistics part is unambiguously the fifth — readers and writers branch on
# the fourth part's emptiness, not on the raw part count.

# _parse_bundle, L50-54
if len(parts) < 3 or len(parts) > 5:
    raise ValueError(f"Expected 3 to 5 parts in parquetbundle, found {len(parts)}")

settings   = parts[3] if len(parts) >= 4 and parts[3] else None
statistics = parts[4] if len(parts) == 5 and parts[4] else None

So: ordering is [core, core, core, settings, statistics]; the engine emits a zero-byte placeholder rather than omitting the delimiter; and "branch on emptiness, not the raw count" is the writer's own stated contract — which is exactly what this PR does. The failure mode the comment worries about (silently dropping auto-generated cluster styles) can't occur via that path.

Worth noting the two sides now express the bound in different units — Python validates 3-to-5 parts, this PR validates 2-to-4 delimiters. Same constraint, and this PR has it right, but it's an easy off-by-one for the next reader; might be worth a word in the comment.

If you'd like, the ASSUMPTION block and the console.warn observability guard could both be reduced now that the invariant is enforced upstream — though keeping the warn is defensible as cheap insurance against writer drift. Your call; I've left the loader alone.

Context: #329 covers the docs half of #310 (documenting the optional statistics table in data-format.md). I've deliberately not touched bundle.ts, delimiter-utils.ts, bundle-writer.ts or their tests so this PR owns the loader fix outright — no conflict from my branch.

One heads-up: this branch still targets the pre-monorepo layout (app/src/…, services/protspace-prep/…), which has since moved to apps/web/src/… and apps/prep/… on main. A rebase will likely need path fixes.

[peymanvahidi]

Coordination note from #329 (docs consolidation), so this PR does not hit a surprise conflict on its next rebase.

#310 had two halves: documenting the statistics table, and fixing the web loader that rejected 5-part --stats bundles. Since this PR is still a draft, the minimal loader tolerance landed in #390 so #310 could close with the epic:

• packages/core/src/components/data-loader/utils/bundle.ts: now accepts 2 to 4 delimiters and branches on the zero-byte settings slot, so statistics stay at part 5. That is the same mechanism this PR uses, and it mirrors _parse_bundle on the Python side. The statistics part is deliberately not sliced or read.
• packages/core/src/components/data-loader/utils/bundle.test.ts: the "reject 5 parts" assertion is inverted, plus a new 6-part rejection.

Rendering the statistics table is untouched and still belongs to this PR.

Suggested rebase resolution: the overlap is only those two files. Take the incoming version of bundle.ts and bundle.test.ts and drop this PR's tolerance hunk, keeping everything additive: projection-metadata*, bundle-writer.ts, delimiter-utils.ts, the control-bar changes, and the prep-service stats step. That leaves this PR as purely "render the statistics", with no functional overlap.

Worth folding into the same rebase: 6 files here are still on pre-monorepo paths.

• app/src/explore/* is now apps/web/src/explore/*
• services/protspace-prep/* is now apps/prep/*

For accuracy: #390 is still open, so this tolerance is not on main at the time of writing.