Auto Camera Registration Pipeline - #1854
Open
romleiaj wants to merge 32 commits into
Open
Conversation
New 'Auto Align (current frame)' button computes the camera-pair alignment automatically using the MINIMA-LoFTR matcher hosted by the VIAME interactive service (companion VIAME branch dev/auto-align), a LoFTR fine-tuned for cross-modality matching (EO<->IR). Flow: the panel asks the platform bridge for an alignment of the selected pair on the current frame; desktop ViewerLoader resolves each camera's image path (same per-camera getters stereo uses) and sends both to the interactive service, which returns a homography plus ~24 spatially-spread inlier correspondences in native pixels. Those are injected as ordinary point pairs (CameraRegistrationStore. applyAutoAlignment): they appear in the correspondences table, are hand-refinable like picked points, fit through the normal homography path, and persist through the existing save and Export-menu downloads. Provenance (autoAlignModel, autoAlignInlierRatio) is stamped into the registration source. Desktop-only, mirroring the text-query wiring end to end: interactive.ts autoAlign -> ipcService alignment-auto-align / alignment-available (matcher weights present in the VIAME install) -> frontend api -> Viewer's provide/inject auto-align bridge (dive-common/use/useAutoAlign) -> RegistrationTools. Web (and installs without the weights) never see the button. Unmatchable scenes surface the service's actionable error (e.g. 'try a frame with more structure') instead of applying a bad warp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
handleAutoAlign gated on loadStereoMetadata(), which rejects anything that isn't a stereoscopic (subType === 'stereo') dataset. A plain multicam rig -- e.g. an EO/IR pair (subType === 'multicam') -- therefore failed with "Auto-align requires a multi-camera dataset" even though deep-feature alignment needs only two cameras' images for the current frame, not a calibrated stereo pair. Factor the (stereo-agnostic) per-camera image-path-getter population out of loadStereoMetadata into populateMultiCamImagePathGetters, and add loadMultiCamMetadata that gates on isMultiCamDatasetMeta (stereo OR multicam). handleAutoAlign now uses the latter. loadStereoMetadata keeps its isStereoscopicDatasetMeta gate unchanged, so the stereo service still never loads on a non-stereo dataset; the getter-building code is moved verbatim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The button creates a camera-pair REGISTRATION -- it automates exactly what the Camera Registration panel does by hand -- so name it with the registration vocabulary (registration = creating the transform, Align View = consuming it). Renames the UI strings, the store method (applyAutoRegistration), the provide/inject bridge (dive-common/use/useAutoRegister), the apispec types (AutoRegisterRequest/Response), the desktop API and IPC channels (auto-register / auto-register-available), the provenance keys stamped into the registration source (autoRegisterModel, autoRegisterInlierRatio), and the feature-plan doc. The interactive service's wire name (command: 'auto_align', VIAME branch dev/auto-align) is deliberately unchanged -- it renames in step with the service side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The injected correspondences were invisible after a successful Auto Register unless the user separately flipped the "Pick points" toggle, because the keypoint layer draws nothing while picking is off. Enable picking in applyAutoRegistration so the points appear immediately for review, matching the summary text's "Review the points" instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
Since Auto Register and registration files can both populate the correspondence table, the toggle's job is broader than placing new points: it gates seeing, dragging, and deleting them too. Comments that referenced the toggle by name follow suit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
applyAutoRegistration merged its matcher provenance (model, inlier ratio) into cameraRegistrationSource, but that stamp is rig-global and written into EVERY per-camera registration file. Auto-registering (or then refining) one pair therefore changed every other camera's file body too, so saving after touching only UV points warned it would overwrite the IR file as well, and the Source readout blended the producer's lineage with matcher details. Drop the stamp: the pair's divergence from a loaded producer registration already surfaces through the refined-from-source warning, like any in-app refit. Persisted per-pair matcher provenance is deferred until the file format has a pair-level source field; the plan doc now records that decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
The divergence note is a persistent status by design (the pair really has diverged from the producer registration, before and after saving), but its copy always opened with "Save, then download...", which reads as a pending to-do even right after a save. Key the action hint on the dirty flag: while unsaved it still says save first; once saved it goes straight to the export instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
Its design rationale now lives in code comments and the PR description; the status/history sections were already stale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
Replace the Promise<any> return and duplicated inline request shapes with AutoRegisterRequest/AutoRegisterResponse from apispec, plus a local interface for the service's snake_case wire payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
The current-frame qualifier moves to a tooltip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
The count includes the reference camera, which is ready by definition but was never registered to anything -- a fresh 3-camera rig read "1/3 cameras registered" before any work happened. Applies to the panel summary, the Align View tooltip, and the import menu hint, which share the same registrationProgress semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
A correspondence now belongs to a (camera pair, image pair), not just a
camera pair. The store's canonical state becomes per-pair lists of
observations -- image-pair identity (image names; frame indices resolved
from them at load time via a viewer-published resolver), an enabled flag,
per-observation producer provenance ('manual' vs a matcher id, closing
the gap applyAutoRegistration documented), optional producer stats, and
the points themselves. Fits pool points from every enabled observation
instead of trusting one frame, and new selectors (framesForPair,
correspondencesForFrame/ForCameraFrame, pairFitStats, setFrameEnabled,
clearFrame) expose the per-frame structure and per-frame reprojection
RMS against the pooled fit.
The registration file format moves to version 2: points live only in
per-observation rows (imageLeft/imageRight identity, enabled, source,
stats) -- no flattened duplicate -- and every loader rejects any other
version outright, because a pre-v2 file would otherwise load as a
matrix-only pair with its points silently dropped. All three persistence
surfaces move together: the client parsers (store + desktop backend's
fromRegistrationPairs), the server cameraCorrespondences meta schema,
and the web/desktop import, export, and seed paths.
mergeRegistrationValues now merges at observation granularity, so a
fresh pipeline result updates the image pairs it covers without
discarding hand-picked observations it doesn't.
The keypoint layer renders only the observations belonging to the image
pair each pane currently shows (each side resolved in its own camera's
frame space), fixing points floating over features that aren't there --
including for hand-picking across frames, which was broken independent
of auto-register. No migration path: v1.9.12's flat format never reached
users, and KAMERA adopts v2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PcZ1d5tifDtgzx3QFbaaFf
The panel becomes two-tier: a Registration Frames list (one row per observation with its point count, per-frame reprojection RMS against the pooled fit as a colored agreement dot, an enable checkbox that excludes a frame from the fit without deleting its points, jump-to-frame, per-frame delete, and prev/next-frame navigation) above a correspondence table scoped to the frame currently being viewed. The transform status reads 'fit from N frames / M point pairs — rms X px', so a frame that disagrees with the consensus is visible and removable instead of quietly dragging the solution. Unresolved observations (images not in this dataset) list with a warning row; producer-skipped candidates show their machine-readable reason. The Timeline work-area gains a marker row for the active pair's registration frames -- positioned with the exact pixel mapping the playhead uses, clickable to seek, dimmed when excluded -- rendered as siblings of the playhead so it stays visible in all four chart modes. ControlsContainer feeds it from the injected registration store, gated on the Camera Registration panel being open, in the Timeline's own selected-camera-local frame space. Opening the panel no longer soft-collapses the bottom controls (the marker row is most useful exactly then); the left sidebar still collapses as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcZ1d5tifDtgzx3QFbaaFf
CameraRegistrationStore.spec exercises the new spine: observation identity stamping (image pair + manual provenance, frame:// pseudo-names without a resolver), per-frame grouping, resolver-driven frame resolution (names win over stale indices; unresolvable images go null, not guessed), per-side correspondencesForCameraFrame lookups, pooled fitting across frames, enable/disable toggles refitting without deleting points, pairFitStats flagging a corrupted frame's disagreement, observation-granular applyRegistrationResult merging (a matcher re-run replaces its own prior result, manual work survives), v2 round trips carrying enabled/source/stats, and the version guard rejecting anything that isn't version 2. cameraRegistrationFiles.spec covers the v2 serialization (observations only, no flattened points, advisory frame omitted when unresolved) and observation-granular merging. Fixtures in the seed/import/discovery specs move to v2 bodies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcZ1d5tifDtgzx3QFbaaFf
… path
The Auto Register button becomes 'Auto Register Frames…': a dialog with
the knobs worth changing (time-bin budget, candidates per bin, min
inliers, pairs topology on a triplet, replace-vs-merge for prior matcher
results) that launches utility_align_cameras_{2,3}-cam over a stratified
candidate spread. DIVE picks for diversity and synchronization -- equal
time bins across the flight, ranked within each bin by inter-camera
timestamp skew when per-frame timestamps exist (100 ms of desync at
survey speed is ~5 m of ground motion RANSAC cannot reject) -- and the
VIAME process picks for image quality within each bin, so oversampling
then pruning keeps the flow single-pass.
Plumbing: runtimeParams.imagePairs carries the per-camera frame subset
as image names (frame://N pseudo-names for video); the desktop multicam
arg writer honors it when writing input{i}_images.txt, keeping the row
ordering contract -- and applies frameRange for multicam image lists,
fixing a pre-existing silent no-op. Video cameras get their selected
frames extracted to stills first, so the register pipes never need
vidl_ffmpeg. On completion, viame.ts sniffs *registration*.json in the
job work dir (substring match, like the calibration hook), maps
extracted-still names back to frame://N identities, and merges the
result into the dataset's registration at observation granularity via
the shared writers -- rejected candidates arrive disabled with their
skip reason, so the frame list doubles as the post-run review. The panel
polls the persisted meta for the result (no cross-platform job event),
never clobbering unsaved edits without confirmation, and shows a
client-computed triplet loop-closure readout in the rig status block.
Availability is now 'is the align pipe in the pipeline list' -- truthful
by construction since the add-on pack installs pipes and weights
together -- replacing the ckpt-presence IPC probe. The single-frame
service path is deleted end to end: useAutoRegister, the AutoRegister*
apispec types, the register-images IPC pair, and
InteractiveServiceManager.autoRegister. The VIAME stdio backend itself
stays (it shares alignment_core).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PcZ1d5tifDtgzx3QFbaaFf
build_multicam_kwiver_settings honors runtimeParams.imagePairs: when a camera has a registration frame subset, only those images are written to its input list, in order, preserving the row-pairing contract (image-sequence cameras; a video subset raises a clear error rather than silently processing the whole clip). run_pipeline pushes the rig's camera names to the register process, and on completion uploads the registration JSON to the dataset folder for provenance and merges it into the dataset's saved registration meta server-side -- registration_output.py mirrors the client's observation-granularity merge (same-identity observations replaced, hand-picked ones kept, matrix-only pairs replacing wholly) and PATCHes the allowlisted meta fields, so the panel's poll-for-result picks it up exactly like on desktop. The client web runPipeline path passes runtimeParams through unchanged; no new endpoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcZ1d5tifDtgzx3QFbaaFf
fromRegistrationPairs, the per-camera file round trip, the multicam transform seed, and the discovery fixtures all move to v2 bodies (observations with image-pair identity instead of flat points). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcZ1d5tifDtgzx3QFbaaFf
camMap's key order is insertion order: cameras are added one at a time inside an awaited per-camera load loop, and entries can survive a dataset switch, so it is not a dependable statement of rig order. Anything where "which camera is first/last" carries meaning -- the registration reference camera, the direction a loop-closure residual is measured in -- needs the order the dataset actually persists. Add displayOrder, published by the viewer from orderedMultiCamCameraNames at load (and cleared on single-camera datasets so a previous rig's order cannot leak), and orderedCameraNames() to read it back restricted to cameras present, falling back to camMap order when no order has been set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
Cameras drop frames independently, so their local index spaces diverge: a frame number produced against one camera does not name the same capture on another. Anything that reads such a number and then renders or seeks it in a different camera's space is off by however many frames the two have dropped between them. Expose the two translations the aligned timeline already supports: translateCameraFrame maps one camera's local frame to another's for the same capture, and cameraFrameToSlot maps it to the rig-wide slot the frame readout and scrubber count in. Both return undefined when the capture has no frame there, so callers can omit rather than draw somewhere wrong, and both pass through when alignment is not active and the spaces coincide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
Observation frames are camA-local, but the Timeline draws in the SELECTED camera's local frame space. Those two spaces only coincide when the rig drops no frames, or when camA happens to be the selected camera: a rig whose cameras drop independently accumulates an offset, putting every marker a frame or two off the capture it describes. Translate each marker through the aligned timeline, and drop the ones whose capture has no frame on the selected camera -- there is no honest place to draw those. The spec's fixture is the real KAMERA fl09 center_view shape: 277 captures at 1s with RGB missing three of them and IR/UV missing three different ones, so each modality has 274 frames over a timeline that disagrees with its neighbours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
The proposal picked a frame index and reused it for every camera, so on a rig whose cameras drop frames independently the matcher was handed images from different instants and asked to register them. On the KAMERA fl09 fixture -- 277 captures with RGB and IR/UV each missing three different ones -- that mispairs from the first drop onward, and nothing downstream can recover it. Take the candidates from the aligned timeline instead, so "frame i" means capture i across the whole rig, and translate each chosen capture into every camera's own local index before resolving paths. Only slots holding a frame from every camera are eligible: the matcher needs the whole set and the pipeline reads image lists off disk, so a gap has no path to send. Gaps stay in the dataset and stay visible in the viewer, they are simply not candidates. Skew ranking reads its timestamps through the same slots. Datasets with no aligned timeline keep the old positional behaviour, which is all they can support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
"Replace existing" drops the prior matcher observations so the fresh run replaces rather than merges. Those removals are the run's own bookkeeping, but they left the store reading dirty, so the completion path stopped to ask whether to discard unsaved edits -- on every single replace-mode run, about changes the user never made. Mark the store saved again when it was clean before the run, so only genuine hand edits still read dirty. A store that was already dirty stays dirty: those really are the user's. This is the same rule frame resolution follows in CameraRegistrationStore.setFrameResolver. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
The post-run confirm is modal and can sit unanswered indefinitely, while the panel went on claiming the job was matching frames. Set the status to what is actually true before awaiting the answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
Two ends of the same mistake. Observation frames are camA-local, but handler.seekFrame() interprets its argument in the SELECTED camera's local space, so jumping to a registration frame from the panel landed on the wrong capture whenever the rig's cameras drop frames independently -- and the frame numbers printed in the list and in the remove-confirmation disagreed with the frame readout and scrubber, which count global slots, by the same drift. Seek through the aggregate controller's seekCameraFrame with camA named explicitly, so the translation happens in the one place that knows the aligned timeline, and carry a separate displayFrame on each row for the number shown. camA-local stays the identity every panel action works in; only the presentation moves to slot space. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
The triplet-consistency readout was wrong at both ends. It sampled a nominal 1000x1000 grid, which on a 12768x9564 EO frame covers a corner and misses exactly the edge divergence that matters, and it compared the result against a fixed 5 px. The residual lands in the last camera's native pixels, so that fixed threshold silently means something different per rig: the same rig error reads ~7.6x larger against a 4864-wide UV camera than a 640-wide IR one, flagging a rig inconsistent for nothing more than having a large last camera. Sample over camera 1's actual frame and judge the residual as a fraction of camera 3's width. 0.001 preserves what 5 px happened to mean on a ~5000px camera; on a KAMERA calibration flight a converged 10-100 frame fit sits near 0.05% and a single-frame overfit rig at 0.15%. Both readouts now show the percentage alongside the pixels. Report nothing while either annotator is still coming up: originalBounds starts as a 1x1 placeholder, and read as a real size it collapses the sample grid onto one corner pixel and yields a confident-looking wrong verdict. Take the camera order from CameraStore.displayOrder too -- which camera is first and last is the direction this is measured in, and camMap's insertion order does not reliably say. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
One flat list mixed matched frames with hand-picked ones and repeated the producer name on every row, which is the least useful place to put it: the distinction that matters is per-section, not per-row. Group the rows under Auto and Manual headers, each carrying its own enabled/total count and mean agreement with the pooled fit, and drop the per-row source chip the header now says better. Give each section a clear-all, since removing a matcher run one row at a time was the only way out of a bad run. It removes every observation of that section INCLUDING the skipped candidates the list hides: those are still stored and still travel into the saved registration file, so leaving them would leave invisible cruft the user cannot see or remove. The confirmation says how many of those there are. Skipped rows stay counted rather than listed -- overwhelmingly they are "pruned", the oversampling remainder from proposing candidatesPerBin per bin and keeping the best. They carry no points and support no action. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
The only way to run the matcher was the stratified proposal, which spreads candidates across the whole sequence. That is the right default, but it gives no way to say "these frames" -- the ones with good overlap, or the stretch where the fit is visibly weak. Queue captures from the panel and run the matcher on exactly those. The queue is kept in global aligned-timeline slots, the same space the frame readout shows, so what the user queues is what they were looking at. Slots translate to each camera's local index at launch, same as the proposal; captures a camera is missing are dropped, because the pipeline reads image lists off disk and a gap has no path to send, and a queue where none survive errors rather than silently launching an empty job. max_frames goes out as one bin per queued frame. It is the pipeline's bin budget -- it keeps the best candidate per bin and prunes the rest -- so leaving it at the proposal's budget would prune away the very frames the user asked for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
The frame-count field offered no guidance, and the failure mode at the low end is the kind that hides itself: one frame fits its own points perfectly -- 100% inliers, near-zero RMS -- while producing the least consistent rig of any setting. The quality readouts look their best exactly when they are least trustworthy. Add a hint for the useful range (the pooled fit converges by about 10 frames; beyond that extra frames mainly tighten triplet consistency, at roughly linear cost) and warn below 5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
Catches the branch up over 70 commits of upstream work. Fifteen conflict hunks across ten files, nearly all from renames upstream made under this branch's changes: - JsonMeta -> JsonConfig, loadJsonMetadata -> loadJsonConfig, DatasetMetaMutable -> DatasetConfigMutable, loadMetadata/saveMetadata -> loadConfig/saveConfig, isStereoscopicDatasetMeta -> isStereoscopicDatasetConfig. Applied throughout, including inside this branch's own additions. - CameraCorrespondences vs CameraObservations: format v2 renamed the type and changed its shape (points per image-pair observation, not a flat list per pair). Upstream only added type annotations to tests this branch had already restructured, so the v2 fixtures stand and the stale annotations go. The substantive one is server-side. Upstream split tasks.py into focused modules, leaving a 52-line re-export barrel, while this branch had added the auto-register hooks to the old monolith -- a mechanical merge produced a 1246-line hybrid that both defined run_pipeline and imported it. Took the barrel and ported the hooks into their new home in dive_tasks/run_pipeline.py: imagePairs off runtime params and through to the multicam settings builder, register:camera_names on the command, and the registration-output ingest on completion. Verified: 1021 tests pass across 77 files, and lint is unchanged from both parents (19 pre-existing errors, none new). Required npm install for onnxruntime-web, newly pulled in by main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the ability to automatically register cameras with MINIMA-LoFTR. Changes aggregation to be over multiple frames, rather than a single frame as in the keypoint gui. Updates registration json to hold these per-frame points.