You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
VGGT-Omega reconstruction backend (--mapping vggt_omega). Feed-forward camera
and depth prediction that produces depth, camera-to-world poses, per-frame
intrinsics, and confidence in one forward pass over the whole sequence.
Installed via the optional vggt_omega extra (pulls vggt-omega from GitHub; FAIR Noncommercial Research License, so research-use
only) and requires a GPU plus the gated facebook/VGGT-Omega checkpoint. (#37)
CLI flags --vggt-omega-model-path and --vggt-omega-image-resolution for the
VGGT-Omega backend. Because there is no sliding window, peak GPU memory grows
with the frame count (~6 GB + ~75 MB/frame), so the frame count must be bounded
with --fps and --begin/--end. (#37)
LingBot-Map reconstruction backend (--mapping lingbot_map). Feed-forward streaming camera and depth prediction that produces depth, camera-to-world
poses, per-frame intrinsics, and confidence. It streams
frame-by-frame against a bounded KV cache and offloads per-frame predictions to
CPU, so peak GPU memory stays roughly constant as the sequence grows. Installed
via the optional lingbot_map extra (pulls lingbot-map from GitHub;
Apache-2.0) and requires a GPU plus the public robbyant/lingbot-map checkpoint. (#37)
CLI flags --lingbot-map-model-path, --lingbot-map-mode (streaming or windowed), --lingbot-map-keyframe-interval, --lingbot-map-window-size, --lingbot-map-overlap-keyframes, and --lingbot-map-attention (auto, sdpa, or flashinfer) for the LingBot-Map backend. (#37)
--no-refine-intrinsics-from-mapper to force the calibrated camera profile K
for backends that would otherwise default to their own estimate (see Changed). (#37)
Demo-style point-cloud cleanup shared across every backend: a relative
depth-edge filter (--depth-edge-rtol, default 0.03) zeroes confidence on
depth discontinuities so object silhouettes no longer smear into the cloud when
viewed from the side, and a sequence-wide confidence percentile cut
(--confidence-percentile, default 20) drops the least-confident points.
Both are ported from the VGGT-Omega/LoGeR demos, apply to the semantic,
geometry-only, and live-preview clouds, and are available on both reconstruct
and view. Set either flag to 0 to disable it. (#37)
CoralscapesV2 segmentation models: coralscapesv2-vit-l-dpt, coralscapesv2-vit-b-dpt, coralscapesv2-segformer-b5 (39 classes) and the
fine-grained coralscapesv2-vit-l-dpt-95, coralscapesv2-vit-b-dpt-95, coralscapesv2-segformer-b5-95 (95 classes). All resolve to public EPFL-ECEO/* repositories; the DINOv3 variants still load the gated facebook/dinov3 encoder and require Hugging Face authentication. (#36)
95-class classes file configs/classes_coralscapesv2_95.yaml (names, colors,
and roles from the CoralscapesV2 dataset), shipped as a packaged resource. (#36)
Each segmentation model declares its own default classes file, so --classes
is now optional and defaults to the model's classes file (95-class models pick
the fine-grained YAML automatically). Pass --classes only to override. (#36)
Changed
Point clouds are now sharper by default: the confidence percentile cut moved
from keeping the top 95% per frame to keeping the top 80% pooled across the
whole sequence, and a depth-edge filter is enabled by default. This drops more
points than before (especially at object edges and in low-confidence frames),
which shifts point counts and benthic-cover fractions. Restore the previous
behaviour with --confidence-percentile 5 --depth-edge-rtol 0 (the percentile
is now pooled globally rather than per frame, so counts will still differ
slightly). (#37)
--refine-intrinsics-from-mapper is now tri-state. Unset, it defaults to on
for backends whose model predicts intrinsics (vggt_omega, lingbot_map) and off for loger, loger_star and scsfmlearner; the resolved value is part
of the mapping cache key. Point clouds from vggt_omega and lingbot_map
therefore shift laterally relative to previous runs, which unprojected with the
camera profile K. (#37)
The vggt_omega and lingbot_map backends no longer return pre-built world_points. Previously they unprojected depth with the model's per-frame
predicted K while reporting the camera profile K downstream, so the cloud,
the viewer frusta and mapping_outputs.npz disagreed on the camera and each
frame carried its own lateral scale (visible as smeared, stacked copies of the
same structure). The orchestrator now settles one K (refined median or
calibrated) and the cloud stage unprojects every frame with it. (#37)
lingbot_map poses are no longer inverted. LingBot-Map's pose_enc decodes to
a camera-to-world [R|t] (its windowed alignment code and demo rely on this),
unlike VGGT / VGGT-Omega whose encoding is camera-from-world. The backend
treated it as camera-from-world and inverted it, so every camera-to-world pose
handed downstream was actually world-to-camera and frames were placed at
mirrored positions, smearing the cloud. Verified on a 49-frame sequence: warping
frame i into frame i+1 with the old poses was no better than not warping at all
(mean abs intensity error 35.2 vs 35.1); with the fixed poses it drops to 27.6,
and sweeping translation scale bottoms out exactly at 1.0, confirming pose and
depth share one scale. vggt_omega is unaffected. (#37)
vggt_omega and lingbot_map run in full fp32 on CPU and MPS. Both models
guard their depth/camera heads with a CUDA-only autocast(enabled=False), so
on other devices the heads inherited the bf16/fp16 autocast context and emitted
quantized depth (visible as concentric depth terraces) and poses. CUDA behaviour
is unchanged (bf16 trunk, fp32 heads). (#37)
Default --segmentation model is now coralscapesv2-vit-b-dpt (was coralscapes-vit-b-dpt). This shifts numeric outputs (benthic cover fractions,
labeled point counts) for runs that relied on the default model. (#36)
Fixed
LoGeR peak VRAM drops by the size of the input batch, 1.6 MiB per frame at the
default 504x280 processing size. The backend preloaded the whole resized
sequence onto the GPU, but Pi3.forward already moves each sliding window to
the model's device itself, so only the window being decoded ever needed to be
resident. Measured on an RTX 3090 over 600 frames: 12561 MiB to 11587 MiB peak
allocated, a 974 MiB saving, with inference time unchanged (59.5 s to 58.8 s).
Outputs are byte-identical, checked with a checksum over depth, poses, points
and confidence. (#35)
The bundled Hero 12 camera profile declares the name it resolves under, gopro_hero_12. It declared gopro_hero_12_4k_wide, so the loaded profile's
name disagreed with the --camera-profile key used to load it. (#39)