[Quality][P0] Add deterministic reference-scene visual and GPU performance qualification

[proggeramlug]

Parent: #126
Related: #24

Problem

Bloom has useful pieces—golden tests, tools/bloom-reference, tools/cycles_reference, realtime examples, and tools/validate.sh—but not one repeatable qualification workflow. tools/validate.sh is report-oriented, reference cameras/configuration are not a versioned contract, and performance measurements can be hidden by vsync or warm-up behavior.

Without a stable corpus, “UE-quality,” “looks better,” and “costs 1 ms” are not independently verifiable claims.

Outcome

A deterministic command and machine-readable result format that compares realtime Bloom output against approved baselines/reference renders across fixed scenes, cameras, quality tiers, motion sequences, and adapters.

Required design

Manifest

Add a versioned manifest (for example tools/quality/scenes.toml) with, per case:

• scene asset path and content hash/version;
• camera transform/FOV and optional deterministic camera animation;
• resolution, warm-up frames, measured frames, seed, timestep, render scale, and quality tier;
• features required/forbidden;
• reference image path or generation command;
• thresholds for SSIM plus a perceptual metric such as FLIP, and targeted masks/metrics where global scores are misleading;
• CPU/GPU/VRAM budgets and the baseline machine class used for hard gates.

Corpus

Start with:

• examples/pbr-spheres for BRDF/material lobes;
• examples/sponza for interior GI, shadowing, leakage, and temporal behavior;
• examples/bistro for scale, sun/sky, streaming pressure, and exterior materials;
• DamagedHelmet.glb for canonical glTF material fidelity;
• one moving skinned + alpha-tested case;
• one 10k+ draw/meshlet and many-light stress case.

Every scene needs at least one still pose and relevant features need a short motion sequence. Assets that cannot live in Git must be fetched by content hash with a license/source manifest.

Runner and outputs

• Run uncapped with fixed timestep; exclude startup, shader compilation, and warm-up frames from steady-state timing.
• Capture named render-graph intermediates through a single debug-capture API.
• Produce result.json, final/intermediate PNGs, heatmaps, and a compact HTML/Markdown summary.
• Record commit, dirty flag, build profile, OS, adapter, driver, backend, capability tier, resolution, and all renderer settings.
• Exit non-zero on hard threshold failure. Support --report-only explicitly for local exploration.

Baseline governance

• Baseline updates require a command that writes a review bundle, never an implicit overwrite.
• A PR changing a baseline must include before/after/diff, metric deltas, timing deltas, and a reason.
• Allow backend-specific baselines only when a documented raster/floating-point difference cannot be normalized.

Acceptance criteria

• One documented command runs the quick corpus locally; one runs the full corpus on a hardware runner.
• Re-running an unchanged commit/config on the same machine produces identical metadata and stable metrics within documented noise bounds.
• The harness detects seeded regressions in BRDF energy, shadow placement, GI leakage, motion history, and texture orientation.
• Vsync cannot make a performance regression appear unchanged; per-pass GPU timestamps and CPU frame work are reported.
• CI uploads all evidence on failure and a compact summary on success.
• The full corpus covers high-end desktop plus at least one constrained tier; unsupported features report their chosen fallback.
• tools/validate.sh, Lumen follow-up: benchmark harness for SSGI quality / convergence #24, and existing reference tools are reused or superseded with migration notes rather than left as competing workflows.

Likely files

• tools/validate.sh
• tools/bloom-reference/
• tools/cycles_reference/
• new tools/quality/
• renderer screenshot/debug-output plumbing under native/shared/src/renderer/
• .github/workflows/test.yml or a dedicated scheduled/hardware workflow

Verification

The PR must attach one complete result bundle from each available backend and demonstrate an intentional failure.

Non-goals

• Claiming that one scalar metric captures artistic quality.
• Hard-gating shared GitHub runners on unstable GPU timing.
• Checking unlicensed third-party assets into the repository.

Dependencies

• [Rendering][P0] Restore deterministic path-tracing goldens and oracle correctness #127 for PT-backed reference cases.
• Can begin independently using raster/reference cases.

[proggeramlug]

Implementation progress and Metal qualification evidence (2026-07-23)

Implemented the deterministic qualification stack:

• Versioned seven-case corpus: high/constrained PBR spheres, Damaged Helmet, Sponza, a pinned deterministic 96-mesh Bistro derivative, skinned Fox plus alpha-tested foliage motion, and a 10,240-mesh/192-light stress scene.
• Fixed-step uncapped warm-up/measurement/capture lifecycle. Sustained windows are now 240–300 measured frames after 120–180 warm-up frames; capture/readback is outside the measured interval.
• Native adapter/backend/capability and active renderer-path reporting. Qualified local adapter: Apple M1 Max / Metal / high-end-rt-bindless / hardware ray query / timestamp query.
• One dormant debug-capture API producing final plus hdr-scene, scene-depth, and all three shadow cascades. Every one of the seven current cases produced all five required intermediates.
• JSON plus Markdown/HTML summaries, strict visual and hard-machine performance gating, explicit report-only mode, baseline review/install with stale-review and path containment protection, reproducibility comparison, and five seeded fault controls.
• Scheduled/manual self-hosted Metal and Vulkan CI, a constrained-tier hard gate, compact job summaries, and always-uploaded success/failure evidence.
• Migration documentation: tools/quality is canonical; tools/validate.sh is retained only as legacy macOS exploration.

Measured evidence:

• Reproducibility retry: PASS with identical stable metadata.
• PBR and Helmet final plus all intermediates: byte-identical across repeated runs.
• Sponza final temporal variance: SSIM 0.9999991, luminance RMSE 0.0000724; all intermediates within the stricter reproducibility limits.
• Seeded controls: BRDF energy, shadow placement, GI leakage, motion history, and texture orientation all DETECTED.
• Post-refactor comparison: PBR and Helmet byte-identical; Sponza SSIM 0.9999971 / RMSE 0.0001277.
• Native shared tests: 131 unit pass, 10 ordinary GPU goldens pass, 3 render-target tests pass; 0 failures.
• bloom-diff: 3 tests pass. bloom-reference: 2 tests pass.
• macOS/shared release checks pass.
• FFI validation: 0 failures, 0 warnings across macOS/Linux/Windows/Android/iOS/tvOS/watchOS/web.
• File-line ratchet, actionlint, Python compile/tests, and git diff --check: PASS.

Important current-quality findings are now locked into review evidence rather than hidden: PBR material lobes are largely flat, and the final Sponza/Bistro output has strong temporal/post blur while the captured HDR scene is substantially cleaner. These are quality-roadmap inputs, not claims of UE-level output.

Governance blocker before closing: approved baselines do not yet exist. A no-write initial review bundle is ready at tools/quality/out/baseline-review-current/review.json; it records baseline_state: absent for all seven cases and includes 35 named intermediates. Per the acceptance contract, a human must inspect it and explicitly run baseline-install --approved-by NAME. I have not self-approved or silently installed those pixels.

Cross-backend status: Metal is locally qualified. The Vulkan/RTX 4080 and DX12 hardware evidence remains pending execution on labelled hardware; #140 tracks the broader Vulkan/DX12 RT qualification dependency.

[proggeramlug]

Review UX follow-up: the human-review bundle now has a self-contained gallery at tools/quality/out/baseline-review-human/review.html plus review.md and the authoritative review.json. The gallery shows each proposed final, applicable before/diff evidence, CPU/GPU p95, and expandable named HDR/depth/shadow intermediates. It remains no-write; no baseline has been installed.