Skip to content

[Rendering][P0] Replace Catmull-Rom TAAU with UE-class temporal super resolution #149

Description

@proggeramlug

Parent: #126
Related: #135, #56, #128

Problem

Bloom's current TAA path writes full-output-resolution history, but reconstructs each fractional-resolution current frame with a 5-tap Catmull-Rom filter and then applies a conventional temporal blend/neighborhood clamp. The default renderer shades at 0.75 scale and used a strong final unsharp mask to cover reconstruction softness.

The result is materially better than the former 0.50 default, but the rich NVIDIA Bistro scene still exposes the remaining gap:

  • fine geometry and textures read as pixelated or soft instead of photographic;
  • high-contrast silhouettes can acquire dark/bright sharpen contours that read as cartoon outlines;
  • subpixel detail, thin rails/wires, alpha-tested foliage, normal-map highlights, and specular edges do not retain native-like stability under movement;
  • raising render scale to 1.0 helps image quality but costs the frame time the reconstruction system is supposed to save.

This is not generally caused by low-resolution source art. Bistro's authored texture set is predominantly 1K-4K, is mipmapped, trilinearly filtered, and uses 16x anisotropy on capable hardware. The missing component is a real temporal super-resolution/reconstruction contract comparable in goals to UE TSR, not a stronger sharpen filter.

Current measured baseline

Apple M1 Max / Metal, rich 551-unique-mesh Bistro profile, 1600x900 output, 0.75 render scale, quality preset 3, 60 warm-up + 120 measured uncapped frames:

  • wall mean: 18.78 ms
  • CPU mean / p95: 8.53 / 30.91 ms
  • timestamped GPU sum mean / p95: 56.56 / 63.55 ms (sum overlaps; use pass deltas, not this value as wall time)
  • TAA pass: 1.58 ms
  • final composite: 1.47 ms
  • main HDR: 9.44 ms
  • bloom: 14.42 ms

The SSGI-stabilized matched run at the same static camera measured wall 19.13 ms, GPU sum 55.94 ms, TAA 1.42 ms, and composite 1.30 ms; run-to-run variance is material, so use multiple alternating pairs for performance claims.

Outcome

A renderer-owned temporal super-resolution path that reconstructs fractional-resolution shading into stable native-output detail without haloing, ghosting, or dependence on aggressive global sharpening. The raylib/Unity-simple contract remains one quality preset or one API call; applications do not author per-pass plumbing.

Do not market the result as UE-equivalent solely because it has a temporal history. Qualify it against native output and motion sequences.

Required reconstruction contract

Resolution and jitter

  • Maintain explicit input/render and output/display extents in every reconstruction calculation.
  • Use a documented low-discrepancy jitter sequence whose sample spread is correct in input-pixel and output-pixel units at every supported scale.
  • Reset or remap history correctly on scale changes, output-size changes, FOV changes, cuts, and projection changes.
  • Choose a texture LOD/mip-bias policy appropriate for reconstruction scale so authored texture detail reaches the input without uncontrolled aliasing.

Reprojection inputs

  • Consume the common [Rendering] Unify temporal history and eliminate TSR, SSR, GI, and PT instability #135 motion convention for static, rigid, skinned, immediate, foliage, transparent, refractive, particle, and procedural producers.
  • Reproject with closest-depth/velocity dilation around silhouettes so foreground motion wins instead of sampling background history.
  • Carry depth plus normal/material identity or an equivalent compact surface discriminator at the resolution needed to reject cross-surface history.
  • Treat sky rotation, camera translation, and infinity-depth content explicitly.

History confidence and locking

  • Track per-output-pixel history confidence/length or an equivalent lock status.
  • Accumulate subpixel detail aggressively only while surface identity and local statistics remain compatible.
  • Shorten/reject history for disocclusion, motion divergence, lighting changes, emissive changes, animated alpha coverage, and newly revealed detail.
  • Keep exposure domains consistent so adaptation does not invalidate/clamp otherwise-correct history.

Reconstruction and anti-aliasing

  • Use source-footprint-aware reconstruction, not a fixed generic upscale kernel alone.
  • Preserve thin geometry, repeated high-frequency patterns, and alpha-tested coverage without moire, breakup, or persistent blur.
  • Bound specular and normal-map shimmer using material/roughness-aware anti-aliasing rather than globally softening the image.
  • Use stable luminance/chroma statistics and clipping that preserve saturated material color without letting stale chroma bleed across edges.
  • Produce stable output at rest and during slow pan, fast rotation, translation, FOV change, and object motion.

Reactive/composition coverage

  • Integrate imported BLEND/transmission, authored reactive coverage, refraction, particles, emissive changes, and custom post content with [Rendering] Unify temporal history and eliminate TSR, SSR, GI, and PT instability #135's reactive contract.
  • Transparent content may not leave history trails or be permanently excluded in a way that aliases badly.
  • Debug output must show missing/invalid motion and reactive producers rather than silently accepting them.

Sharpening

  • Reconstruction quality must not depend on high-strength global unsharp masking.
  • Keep sharpening an explicit final-stage quality knob.
  • Use edge/contrast-aware behavior that does not draw dark or bright contours on high-contrast silhouettes.
  • Do not add a second sharpening pass by default or double-sharpen presets.

Debugging and telemetry

Capture-only views:

  • input-to-output sample footprint and jitter phase
  • motion vectors after dilation and selected velocity source
  • reprojected history UV
  • depth/normal/material rejection reason
  • reactive/composition mask
  • history confidence/length or lock status
  • neighborhood/clipping extent and clamp magnitude
  • current/history blend weight
  • disoccluded/newly revealed pixels

Telemetry:

  • input and output extents, scale, preset, jitter index/spread
  • accepted/rejected/reactive/disoccluded pixel counts and ratios
  • history-length/confidence histogram
  • reconstruction, dilation, and optional sharpening pass timings
  • persistent/transient image and buffer bytes
  • bind-group/pipeline/graph churn after warm-up

Capture-only diagnostics must add zero persistent resources and must be released after readback.

Required corpus

At minimum automate matched native and reconstructed sequences for:

  1. rich Bistro exterior and interior: cobbles, floor tiles, wall texture, window rails, hanging wires/lights, glass, hard facade/sky edges, normal-map highlights;
  2. Sponza: foliage/cards, thin ornament, high-frequency stone, depth-layer disocclusion;
  3. slow subpixel pan and crawl;
  4. fast yaw/rotation and six-metre translation;
  5. camera cut, FOV step, render-scale step, and output resize;
  6. moving rigid and skinned geometry;
  7. animated alpha-tested foliage and reactive transparent/refraction content;
  8. dark interior looking toward bright exterior and emissive on/off;
  9. specular/roughness sweep and minified normal maps;
  10. web/fallback adapter without optional native features.

Store sequences or deterministic per-frame metrics, not only final settled images.

Acceptance criteria

  • At 0.75 input scale, fixed Bistro/Sponza stills are materially closer to native 1.0 than the current Catmull-Rom TAAU baseline by luminance SSIM, RMSE, OKLab delta, and Sobel-edge delta; attach all four metrics and images.
  • No hard-silhouette dark/bright sharpen contour is visible. The existing sharpen fixture must keep mean luma delta at or below 1.0 on at least 100 hard-edge pixels while changing at least 2% of ordinary-detail samples.
  • Slow-pan stable-region mean flicker does not exceed native TAA by more than the approved tolerance, and coherent flicker/moire regions are explicitly bounded.
  • Fast motion/cuts leave no visible stale-history trail beyond the approved reset/recovery window.
  • Thin wires, rails, alpha-tested foliage, and repeated floor/cobble patterns remain continuous and do not alternate between sharp/soft phases.
  • Reactive transparency, refraction, emissive changes, rigid motion, and skinning pass the [Rendering] Unify temporal history and eliminate TSR, SSR, GI, and PT instability #135 producer corpus.
  • 0.75 reconstruction preserves the performance advantage over native 1.0 on the qualification GPU.
  • Multiple alternating frozen-binary A/B pairs show no statistically significant full-frame CPU/GPU regression versus the current 0.75 path; report p50/p95 and per-pass timings. If the reconstruction pass itself grows, offsetting in-scope savings must be real and measured, not inferred from timestamp noise.
  • Steady state adds no per-frame allocation, pipeline creation, bind-group creation, graph compile, or CPU-GPU synchronization.
  • Added persistent/transient memory is fully reported and bounded by quality tier; resize/scale cycling does not leak.
  • Native Metal, Linux Vulkan, and web/fallback paths pass their supported corpus; unsupported capability choices degrade explicitly and truthfully.
  • Public usage stays one preset/API call, with individual expert overrides optional.

Likely files

  • native/shared/src/renderer/shaders/post.rs
  • native/shared/src/renderer/postfx_chain.rs
  • native/shared/src/renderer/formats.rs
  • native/shared/src/renderer/temporal_reactive.rs and temporal diagnostics
  • native/shared/src/renderer/quality_preset.rs
  • native/shared/src/renderer motion/depth/material outputs
  • native/shared/tests/golden_render/temporal_history.rs
  • native/shared/tests/golden_render/quality_presets.rs
  • tools/quality scene/motion orchestration and metrics
  • src/core/index.ts quality/reconstruction API

Non-goals

  • Frame generation.
  • Hiding the issue by forcing native resolution on every preset.
  • Hiding instability with blur, excessive history weight, or stronger sharpen.
  • Requiring game code to hand-wire engine-internal masks and histories.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions