Catalog: fetch browse data and install payloads separately

[onel]

What & why

The box pulled one bulk snapshot (GET /catalog/sync) carrying every published app's verbatim manifest.yml, compose.yml and resolved images map. On a 43-app catalog that is 614KB, of which 77% is install payload for apps the box will never install; the browse data the store actually renders is 106KB. Every box paid the full 614KB on first sync and on every catalog change.

Browse and install are now two fetches:

• GET /catalog?env=<environment> — display records, landing page, category vocabulary, and an opaque version token. Filtered server-side to the box's surface, so the box applies no visibility pass of its own.
• GET /catalog/apps/{id}/manifest and /compose — the verbatim file as application/yaml, fetched only when the box actually installs that app. Load takes a context now.

The index digest is gone. It was recomputed by re-marshalling the parsed app index, which made field order load-bearing and turned any new published field into a flag day: the digest stopped matching, verify() refused the snapshot, and the box showed an empty store. That was measured on external_costs, not theoretical. It closed no threat TLS (origin) and HTTP framing (truncation) do not already close, so version replaces it as a token the box stores, echoes, and never recomputes, and unknown keys are dropped the way encoding/json drops them everywhere else. The schema_version refusal stays.

Installed apps read their own manifest. This is the load-bearing half. Load was never install-only — the detail page's mail picker and the mail rebind pre-check called it for apps already installed — and a live fetch there would put a routine page load behind the catalog service. Both now read life.InstanceManifest, the copy writeInstanceDir already persists. That also fixes a pre-existing bug: an installed app's manifest used to disappear the moment the app was unpublished.

Published URLs are opaque. icon_file / screenshots became icon_url / screenshot_urls, joining manifest_url / compose_url. The box follows what it is given (absolute URLs on another origin included) and assembles nothing, so artwork or documents can move to object storage without a box-side change.

Beyond the issue

Two doc changes ride along at the maintainer's request, rather than as a separate PR:

• contributing.md Step 6/7 swapped. The three docs disagreed: CLAUDE.md and code-review.md put the self-review after the PR opens and make Greptile mandatory; contributing.md put it before and never mentioned Greptile. Greptile only runs on an open PR, so a pre-push review structurally cannot include it. Step 7 now names both halves.
• PR template. The per-app-field checkbox warned that a new field freezes deployed boxes. The digest that caused that is gone, so the warning described behavior the box no longer has.

Spec(s) touched

• docs/specs/APP_STORE.md — banners, # Failure modes, # What we run, # Landing page, # Category labels, # What the box models (rewritten), # Locked decisions.
• docs/specs/DECISIONS.md — new entry 2026-09-04. The 2026-07-02 entry's "TLS + integrity digest" trust story loses its second half.
• docs/specs/NEXT.md, docs/architecture.md — kept in step.
• Progress: docs/progress/catalog-split-browse-and-install.md.

What was tested

• make check green, minus the pre-existing libpam0g-dev gap on my machine (pamverifier / cmd/host-agent-real do not build there; neither is touched by this change). Everything else in go test ./... passes.
• Real Docker + real Caddy, twice, via make dev — not unit tests alone:
  1. From a mkcatalog seed file (the air-gapped lane's path): installed whoami end-to-end, served on whoami.local.
  2. Against a stand-in HTTP endpoint serving the four new routes: brain logged catalog: synced browse payload, endpoint logged BROWSE env='appliance' plus DOC /catalog/apps/whoami/manifest and /compose only at install time, install completed, app served through Caddy.
• New tests: env on the browse fetch, no document fetch while browsing, document fetch on Load, absolute document URLs on another origin, 404-vs-500 document failures, seed-file inline payload, opaque-URL asset caching and containment, empty screenshot slots, unknown-key tolerance, opaque version token.
• bash -n clean on both changed bootstrap scripts.

Known gaps & deviations

• An installed app that leaves the box's surface loses its card icon. The issue offered two answers — persist the display record, or accept and document the degradation — and this takes the second. The install is untouched (manifest and compose live next to it); only catalog-supplied display metadata degrades, and the card falls back to the instance row's own name and version. Documented in APP_STORE.md # Failure modes.
• The real control plane's routes are not exercised here. The new API is being built in the private cloud repo. This side is proven against an in-process fake plus the stand-in endpoint above; the two-repo seam is a coordinated change and cannot be checked by reading this diff. The cloud side must land before this can ship — a box on this code against the old endpoint parses a payload with no document URLs and fails at install.
• No boot-lane run. dev/test-qemu / dev/cloud/test need root + KVM and are not in the normal loop. Both scripts changed only by dropping one flag. Validated on the next CI / Cloud image run.
• An install does four document fetches, not two — installApp loads to validate elections, then Manager.Install loads again. Found in the live run. Correct but wasteful; it is a change to the install call chain, so it is logged as a follow-up rather than widened into this PR.
• Self-review: the agent half ran pre-push under the old Step 6 and raised two Block findings, both fixed with tests (assetCachePath did not contain the app id — a hole the old safeJoin had too; detailOfApp emitted URLs for empty screenshot slots). Greptile's half is still pending on this PR and I will work it next.

Platform gaps

None.

Closes #434

[greptile-apps]

Confidence Score: 5/5

The changes since the previous review appear safe to merge, with the outstanding Docker-live compilation issue fully fixed.

The non-tagged mustLoadApp helper returns the CatalogApp required by the current Install signature and is available to every updated Docker-live caller. The added tagged vet passes do not introduce a new native dependency into the standard gate, and no new actionable failures were found.

Important Files Changed

Filename	Overview
internal/lifecycle/dockerlive_test.go	All six tagged test callers now load and pass a CatalogApp, fully resolving the previous compilation finding.
Makefile	The vet target now type-checks each supported tagged test variant without adding a new environment dependency.
docs/progress/catalog-split-browse-and-install.md	Records the resolved review findings and the new tagged-vet safeguard.

_{Reviews (3): Last reviewed commit: "Fix the dockerlive Install callers, and ..." | Re-trigger Greptile}

[onel]

Greptile P1 — catalog payload loaded twice — is a real bug and is fixed in 0e5f192.

You are right about the consequence, and it is worse than the wasted round trip I had recorded it as. While the manifest and compose lived in one bulk snapshot, the job's re-read was cheap and near-enough consistent. Now each is its own fetch, so a republish landing between the request's load and the job's load means the job installs a payload nothing validated: a folder mount the elections still name but the new manifest dropped, a newly required config field the installer was never asked for, or one publication's manifest paired with another's compose.

The fix keeps one version-consistent pair for the whole install:

• lifecycle.CatalogApp carries the parsed manifest and the verbatim compose as one value.
• Manager.Install takes that pair instead of a manifest id, so it installs byte-for-byte what the caller validated.
• LoadCatalogApp is called exactly once, in the request, and the pair is captured by the job closure.
• TestInstallUsesTheLoadedPayloadNotAReFetch republishes the catalog between the load and the install under a different name and slug, and asserts the install ignores it.

One part I have not closed, deliberately. The two documents of a single payload are still two HTTP fetches, so a publish landing between them can still pair one publication's manifest with another's compose. Closing the request/job window does not close that one — it needs the payload addressable as a unit from the control plane (a version on the document routes, or an ETag the box requires to match across both fetches). That is a contract change in the cloud repo, so it is recorded as a two-repo follow-up in the progress entry rather than patched here.

For the record: I had seen this exact code path during an inner-loop run, counted the four fetches, and filed it as an efficiency follow-up. Good catch on what it actually was.

[greptile-apps]

Dockerlive Tests No Longer Compile

Install now requires a CatalogApp, but all six callers in dockerlive_test.go still pass string IDs. As a result, the documented go test -tags dockerlive ./internal/lifecycle/ workflow fails to compile, preventing the live Docker provisioning suite from running. Update those callers to load and pass a CatalogApp, as the other lifecycle tests now do.

[onel]

Second-pass finding is right, and it is a regression my own fix introduced — fixed in cffaa78.

The CatalogApp signature change left the six Install callers in internal/lifecycle/dockerlive_test.go passing a string id. Two things let that through:

• go vet ./... does not compile a file behind a build tag. It is excluded from the build, so make check was green across the whole change. The gate could not have caught it.
• My sweep of the call sites missed it. I rewrote ~55 Install callers with a regex anchored on context.Background(); this file passes a ctx variable, so it did not match — and nothing then failed to compile to tell me.

Fixing just the file would leave the hole open for the next signature change, so make vet now vets the tagged variants too:

VET_TAGS := dockerlive usermgrtest avahitest nmtest pamtest

That is cheap and belongs in the gate: vet only type-checks, so it needs none of the hardware those tags name. Actually running those tests still needs the real system each tag stands for, per TESTING.md — unchanged.

Verified locally with go vet -tags <tag> for dockerlive, usermgrtest, avahitest and nmtest (all clean); pamtest needs libpam0g-dev, which this machine lacks and CI has.