Skip to content

v0.5.0

Choose a tag to compare

@rsasaki0109 rsasaki0109 released this 27 Sep 01:51
· 573 commits to main since this release
0b38154

Added

  • Phase 39 adds the paper-faithful Frequency Consistency Metric (FCM) to
    image-evaluate and rendered-evaluate: grayscale 5x5 LoG (sigma=1,
    zero padding), normalized Frobenius error clamped to [0, 1], and the
    lower-is-better max_frequency_consistency gate for kind: image and
    kind: rendered. JSON metadata records the flat-reference policy (both
    flat = 0; candidate non-flat = 1), and reports/triage/PR comments surface
    the aggregate mean.

  • ca uncertainty-evaluate and kind: uncertainty for position NEES (DoF=3),
    normalized position NEES, and chi-square coverage from GT plus validated 3x3 state
    covariance. NIS remains deferred because innovations are not available.

  • Optional DreamSim perceptual distance for image-evaluate and
    rendered-evaluate, including max_dreamsim_distance config gates,
    triage, PR-comment deltas, and a [perceptual] dependency extra. The first
    real invocation may download official pretrained weights.

  • Experimental reference-free ca plane-consistency evaluation and
    kind: structure CI checks. The deterministic
    plane_normal_dispersion / coplanar_offset_rmse metrics are
    PNE/CPV-inspired local-plane proxies, not claims of reproducing the
    published metrics; they surface in gates, triage, JSON, and PR comments.

  • Benchmark report bundle golden path — ca benchmark eval --out <dir>
    now writes a reproducible report bundle for CI artifacts and static
    publishing: metrics.json, summary.md, report.html,
    manifest.lock.yaml, and provenance.json. The bundle records the
    suite, gate, input hashes, CloudAnalyzer version, and portable paths so
    a SLAM/LIO candidate map + trajectory can be reviewed without relying on
    machine-local state. The README now leads with this "dataset/reference +
    candidate outputs -> metrics/report/gate" workflow, and
    .github/workflows/slam-benchmark-smoke.yml dogfoods the clone-after-install
    smoke path.

  • Static benchmark leaderboard builder — ca leaderboard build <bundle>... --out <site> turns one or more benchmark report bundles into
    a self-contained results.json + index.html site. Rows include dataset,
    sequence, method, gate result, CloudAnalyzer version, report links, artifact
    hashes, and metric values; incomparable rows are surfaced as warnings rather
    than silently mixed.

  • Unified gate severity policy — config-driven ca check now shares
    a stable gate summary schema across artifact, trajectory, run,
    loop-closure, image, and rendered checks. Check specs can use
    severity: fail | warn | soft_fail | skip | not_applicable, and the CLI
    exposes --warn-only and --strict with consistent exit-code behavior for
    CI.

  • SLAM driver conformance helper — plugin authors can call
    run_slam_driver_conformance(MyDriver) from ca.testing.conformance to
    validate metadata, deterministic output paths, trajectory/map shapes,
    stdout/stderr discipline, failure behavior, and writer compatibility before
    publishing a driver adapter.

  • image config check kind (Phase 31) — wires ca image-evaluate
    into the config-driven QA layer so photometric quality can ride the
    same regression gate as every other artifact. A kind: image check
    takes rendered_dir + reference_dir, scores PSNR / SSIM per pair,
    and gates on the aggregate mean via gate: { min_psnr, min_ssim }
    (min_psnr in dB). Bit-identical pairs (PSNR = +∞) are excluded from
    the mean and trivially pass min_psnr; a gate fails when zero pairs
    match across the two directories. The kind emits Markdown/HTML reports
    (save_image_report), participates in ca's severity-weighted triage
    ranking, and surfaces PSNR / SSIM means with up/down deltas in
    ca report-pr-comment. This is the first non-geometric axis to reach
    the CI gate; LPIPS (an optional [photometric] extra, adding a
    max_lpips gate key) follows in a later phase. See docs/ci.md
    § Config-Driven QA.

  • ca image-evaluate (Phase 30) — first photometric evaluation
    surface in CloudAnalyzer. Scores a set of rendered images against a
    reference (ground-truth) set on PSNR + SSIM, pairing by filename
    across two directories. Implementation is pure numpy +
    scipy.ndimage.gaussian_filter (no new heavy dep). Emits per-pair
    metrics plus a per-metric summary (mean / median / min / max). New
    module ca/core/image_evaluate.py carries the
    ImageEvalRequest / ImageEvalResult contract and the psnr /
    ssim helpers. This is the foundation for a future
    ca rendered-evaluate that would drive a 3DGS PLY into images at
    known camera poses and chain into this scoring path; see
    docs/commands/image-evaluate.md for the full reference.

  • Canonical third-party SLAM driver plugin (Phase 29). The repo
    now ships
    plugins/cloudanalyzer-driver-example/,
    a complete pip-installable package that registers an Open3D
    scan-to-scan point-to-point ICP driver under the
    cloudanalyzer.slam_run_drivers entry-point group. After
    pip install -e plugins/cloudanalyzer-driver-example,
    ca slam-run --driver example works end-to-end. The plugin depends
    only on what CloudAnalyzer already pulls in (no kiss-icp / kiss-slam
    / small_gicp extras required) — so it doubles as a worked template
    for third-party authors. CI installs and exercises it via a new test
    (tests/test_slam_run_plugin_example.py), so the entry-point
    pathway introduced in Phase 28 is now validated against regressions
    by a real package install, not just by in-process monkey-patches.

  • Plugin / entry-point registry for ca slam-run drivers (Phase 28).
    ca.core.slam_run now exposes register_driver(name, factory),
    get_driver(name), and list_drivers(). The three built-in drivers
    (kiss-icp, kiss-slam, small-gicp) register themselves at module
    import time via lazy factories. Third-party packages can publish their
    own driver under the cloudanalyzer.slam_run_drivers entry-point
    group and have it surface automatically as
    ca slam-run --driver <their-name> after pip install. The CLI's
    former hard-coded if/elif over driver names is gone — --driver
    now resolves through the registry. Broken third-party plugins are
    logged and skipped instead of crashing the CLI. See
    docs/commands/slam-run.md § Adding a third-party SLAM driver for
    the contract + a worked pyproject.toml example.

Changed

  • Public demo/report artifacts now scrub machine-local absolute paths and
    escape embedded copy-command strings consistently, making checked-in HTML
    reports safer to publish and deterministic in CI.
  • The synthetic-room 3DGS fixture check in the main Test workflow now uses
    --skip-renders, keeping CPU GitHub runners on the geometry fixture path
    while the rendered self-QA workflow continues to dogfood kind: rendered
    with checked-in reference PNGs.
  • ca slam-run --driver help text now lists the three built-in drivers
    and points at the entry-point group; the validated choice set is
    computed dynamically from the registry rather than hard-coded.

Fixed

  • Avoided a circular import on script entry points that imported
    ca.batch -> ca.info -> ca.core -> ca.core.checks -> ca.batch by loading
    get_info lazily inside batch_info.
  • Fixed .github/workflows/leaderboard.yml parsing by replacing a YAML-hostile
    shell heredoc commit message with normal git commit -m ... -m ... flags.
  • Fixed oriented 3D box IoU (ca detection-evaluate /
    ca tracking-evaluate) on NumPy 2.x, which rejects 2D inputs to
    np.cross; polygon clipping now uses an explicit 2D cross product.
  • Capped open3d<0.20: open3d 0.20.0 requires libusb-1.0 at import time
    and ships stubs that reject str paths.