[Feature Request] Image-space (photometric) NeRF / radiance-field training via the facade

[ooples]

Problem

Real NeRF training (per the original paper and every reference implementation — nerfstudio, PyTorch NeRF, JAX NeRF) uses a photometric loss: rendered pixels vs photo pixels, with the gradient flowing backward through the volume-rendering integral into the NeRF's parameters.

AiDotNet's NeRF<T>.Train(Tensor<T> input, Tensor<T> expected) accepts input of shape [N, 6] (position + view direction) and target of shape [N, 4] (rgb + density). That's per-point supervision — the caller must already know the ground-truth (rgb, density) at each sampled 3D point.

Consequences:

1. You can't actually train NeRF on photos. You can only feed it point-cloud-style tuples derived from an analytical or LIDAR-scanned scene. That's not what NeRF is in the paper.
2. The MLP has no incentive to learn sharp density boundaries where it matters most (surfaces). Empirically demonstrated: Week 11 Session B demo trained paper-scale NeRF (192-wide × 8 layers, 3000 epochs, 32k point samples across the full render bbox) and NeRF converged to smooth-blob density everywhere instead of sharp cube surfaces. Per-point MSE doesn't weight surface pixels the way volume-rendering integration does.
3. XML documentation on IRadianceField<T> describes NeRF as "photos + camera poses → walkable scene." Users following the docs hit an API wall.

Reference-implementation landscape

Framework	API shape	Level
PyTorch NeRF (Mildenhall reference)	User writes ray sampling loop manually + calls render(rays_o, rays_d, model) + optimizer.step()	Low — primitives only
JAX NeRF (Google)	Same as PyTorch NeRF	Low
Instant-NGP (tiny-cuda-nn)	trainer.train_from_images(images, poses) — one convenience call	Mid
nerfstudio	trainer = Trainer(config); trainer.train() — YAML/config-driven full pipeline	High

AiDotNet's design principle ("industry-standard defaults + full customization") points at mid-level with primitives exposed underneath: one convenience call that Just Works, plus underlying RenderRays / SamplePointsAlongRays / QueryField public methods for callers who want to build custom loops.

Proposed direction (agreed in design discussion)

Loader integration — refactor IDataLoader<T> → IDataLoader<TInput, TOutput>

The cleanest type-theoretic fit for image-space training is a generic IDataLoader<TInput, TOutput> where the image path is IDataLoader<ImageView<T>, PixelBatch<T>> (or similar). This matches how PyTorch's DataLoader is generic over the collate output type. No fake pair semantics, no polymorphism games, no new Configure* method — just a wider type shape on the existing interface.

Migration implications:

• Every shipped loader (InMemoryDataLoader<T>, LeafFederatedDataLoader<T>, GraphDataLoader<T>, etc.) needs to migrate to IDataLoader<Matrix<T>, Vector<T>> (or their natural TInput/TOutput shape).
• ConfigureDataLoader signature widens; call sites unchanged for callers using the shipped loaders + factory helpers.
• Consumers that reference IDataLoader<T> directly need to update their generic param list.
• Recommend one atomic PR migrating the interface + all shipped loaders + facade wiring together; ship an [Obsolete] IDataLoader<T> alias for one release cycle to give community consumers time to update.

IImageTrainable<T> — model marker for image-space training support

public interface IImageTrainable<T>
{
    // Called per training step by the facade. Model handles ray sampling from
    // the view set, volume rendering, loss computation, and gradient application.
    // Returns the batch loss for the caller's telemetry.
    T TrainOnImageBatch(IDataLoader<ImageView<T>, PixelBatch<T>> views, IOptimizerOptions<T> optimizerOptions);
}

NeRF<T>, InstantNGP<T>, and GaussianSplatting<T> all implement IImageTrainable<T>.

ImageView<T> + ImageTrainingDataLoaders (shipped implementations)

public sealed class ImageView<T>
{
    public Tensor<T> Photo { get; init; }     // [H, W, 3], any H/W per view
    public Vector<T> CameraPosition { get; init; }
    public Matrix<T> CameraRotation { get; init; }
    public T? FocalLength { get; init; }      // nullable — null triggers auto-detect
    public LearnedPrior<T>? Prior { get; init; }  // optional single-image/few-shot prior
}

public static class ImageTrainingDataLoaders
{
    public static IDataLoader<ImageView<T>, PixelBatch<T>> FromViews<T>(IEnumerable<ImageView<T>> views);
    // Convenience: auto-loads photos + poses from a directory with a COLMAP transforms.json
    public static IDataLoader<ImageView<T>, PixelBatch<T>> FromDirectory<T>(string path);
}

Facade wiring

ConfigureDataLoader (widened signature) + ConfigureModel + ConfigureOptimizer — no new Configure* methods. Facade branches at BuildAsync:

if (loader is IDataLoader<ImageView<T>, PixelBatch<T>> imgLoader && model is IImageTrainable<T> imgModel)
{
    // Image-space training loop — model owns ray sampling + volume rendering + backprop
}
else if (loader is IDataLoader<ImageView<T>, PixelBatch<T>> && !(model is IImageTrainable<T>))
{
    throw new InvalidOperationException(
        "Image training loader supplied but model does not implement IImageTrainable<T>. " +
        "Use a radiance-field model (NeRF, InstantNGP, GaussianSplatting) or provide a " +
        "regular IDataLoader<TInput, TOutput> for per-point supervision.");
}
else
{
    // Existing supervised path
}

Two opt-in mechanisms (loader type via TInput/TOutput generic parameters + model interface) so custom loaders (video frames + IMU poses, LiDAR-derived views) OR custom models (community radiance-field forks) can plug in independently.

Beyond-industry excellence goals

None of the reference impls do these cleanly; all four are goals for this feature:

1. Auto-detect focal length from image (EXIF or fallback)
   ImageView<T>.FocalLength is nullable. Loader reads EXIF from the loaded photo when available; falls back to a reasonable default based on image dimensions if not. Reference impls all require caller to pass focal explicitly.

2. Mixed-resolution photos in one view set
   Nerfstudio requires all photos pre-resized to a common resolution. AiDotNet samples rays proportionally per-photo (each view's H×W ray budget scales with its pixel count relative to the median). Callers drop in raw photos without preprocessing.

3. Auto scene bounding box + near/far from pose set
   Nerfstudio wants manual near/far in the config. AiDotNet estimates from the camera poses: intersect view frusta, tight bbox, near/far to hit the box comfortably from every camera. Callers can override.

4. Progressive coarse→fine sampling + single-image LearnedPrior mode
   Progressive coarse-to-fine ray sampling as a built-in default schedule (coarse samples for first N iters, fine samples once density is coarsely learned). Also: single-image / few-shot reconstruction via optional LearnedPrior<T> on ImageView<T> — unify Zero123/TripoSR-style reconstruction into the same NeRF class with a prior that hallucinates missing views. Prior is fully customizable (industry defaults ship, users swap).

Wiring surface (no new Configure methods)

• ConfigureDataLoader (existing) — signature widens to IDataLoader<TInput, TOutput>
• ConfigureModel (existing) — no changes; model's IImageTrainable<T> is what the facade detects
• ConfigureOptimizer (existing) — options flow to the model's TrainOnImageBatch for per-attribute schedules (per GaussianSplatting facade training silently ignores ConfigureOptimizer settings (LR, betas, weight decay) — uses internal updates instead #1833's routing)

Scope + sequencing

This is a multi-week refactor in isolation. Recommend sequencing:

1. Land the IDataLoader<TInput, TOutput> refactor + [Obsolete] alias for IDataLoader<T> as its own PR (self-contained; unblocks everything downstream). Ship in one atomic PR to keep tests + shipped loaders coherent.
2. Land IImageTrainable<T> + ImageView<T> + ImageTrainingDataLoaders + facade branch as a follow-up PR.
3. Implement NeRF<T>.TrainOnImageBatch first (paper-standard); InstantNGP<T> (hash-grid-accelerated) and GaussianSplatting<T> (splat-space photometric with densification) as follow-up PRs.
4. Excellence goals (EXIF focal, mixed-resolution, auto bbox, progressive + prior) as separate follow-ups once the base image-space path is verified against a canonical Blender scene.

Related

• Facade BuildAsync throws on radiance-field models: feature-selection guard misses 1-D-encoded neural inputs #1826 / fix(facade): unblock BuildAsync for radiance-field models (#1826) #1829 — original facade unblock chain
• ParameterCount cached at construction, stale after lazy shape resolution #1832 — parameter-count cache bug (also surfaces when trying to save trained-on-images models)
• GaussianSplatting facade training silently ignores ConfigureOptimizer settings (LR, betas, weight decay) — uses internal updates instead #1833 — GS optimizer hyperparameter routing (image-space GS training needs the routing hook)
• GaussianSplatting densification (split/prune during training) not exposed through the facade #1835 — GS densification exposure (interleaves with image-space training loop)