feat(viewer): interactive click-to-coordinate with Python bridge

[spomichter]

what

adds click-to-coordinate support to the custom_callback viewer example. click on an entity in a 2D/3D spatial view → world-space coordinates sent via TCP to a Python bridge.

why

DimOS needs click-to-navigate — operators click on a map point, robot goes there. rerun is currently read-only with no backchannel from viewer to application.

uses the official StartupOptions::on_event callback API. zero changes to rerun core crates — everything lives in examples/rust/custom_callback/.

changes

file	what
interaction/protocol.rs	ViewerEvent + AppCommand enums with bincode serde
interaction/handle.rs	InteractionHandle — mpsc channel wrapper, 3 unit tests
interaction/sender.rs	ViewerEventSender — TCP client with auto-reconnect
interaction/mod.rs	module exports
viewer.rs	on_event callback: intercept SelectionChange, debounce (100ms), send via TCP
python_bridge/bincode_codec.py	pure Python bincode encoder/decoder
python_bridge/bridge_bincode.py	TCP server receiving click events from viewer
python_bridge/bridge.py	JSON protocol bridge (dev/prototyping)
python_bridge/ground_plane.py	invisible ground plane for click-anywhere support
build-viewer.yml	CI workflow for linux x64 + macOS arm64

architecture

Rerun Viewer (Rust)
  ↓ SelectionChange event (position: Option<glam::Vec3>)
  ↓ on_event callback
ViewerEventSender (TCP client)
  ↓ length-prefixed bincode, port 8888
Python ViewerBridge (TCP server)
  ↓ @on_click decorator
DimOS RerunInteractionModule → Nav streams

setup & build (Ubuntu 22.04)

1. install rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# installs latest stable (1.93+), rerun requires rust-version = 1.92

2. install system dependencies

sudo apt-get update && sudo apt-get install -y \
    build-essential cmake clang \
    libgtk-3-dev \
    libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev \
    libxkbcommon-x11-dev libxkbcommon-dev \
    libvulkan-dev \
    libwayland-dev

3. clone & build

GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 -b feat/interactive-click-bridge https://github.com/dimensionalOS/rerun.git
cd rerun

# debug build (~15 min first time, 228MB binary)
cargo build -p custom_callback

# or release build (~25 min, smaller + faster)
cargo build --release -p custom_callback

4. run tests

# rust tests
cargo test -p custom_callback
# expected: 5 tests pass (4 unit + 1 doctest)

# python bridge tests
cd examples/rust/custom_callback/python_bridge
python3 test_bridge.py -v          # 4 tests
python3 test_bridge_bincode.py -v  # 6 tests

e2e test (requires display)

terminal 1 — python bridge

cd rerun/examples/rust/custom_callback/python_bridge

python3 -c "
from bridge_bincode import ViewerBridge
from bincode_codec import ClickEvent

bridge = ViewerBridge(port=8888)

@bridge.on_click
def on_click(event):
    print(f'CLICK at ({event.position[0]:.3f}, {event.position[1]:.3f}, {event.position[2]:.3f})')
    print(f'  entity: {event.entity_path}, view: {event.view_id}')

print('bridge listening on port 8888...')
bridge.start(blocking=True)
"

terminal 2 — viewer

cd rerun
./target/debug/custom_callback_viewer
# or ./target/release/custom_callback_viewer

should see in terminal 1: Viewer connected from ('127.0.0.1', ...)

terminal 3 — log test data

option A: entity spheres only (click on visible objects)

pip install rerun-sdk numpy

python3 -c "
import rerun as rr
import numpy as np

rr.init('click_test', spawn=False)
rr.connect_grpc('rerun+http://127.0.0.1:9877/proxy')

grid = np.linspace(-5, 5, 20)
xx, yy = np.meshgrid(grid, grid)
positions = np.stack([xx.flatten(), yy.flatten(), np.zeros_like(xx.flatten())], axis=-1)

rr.log('world/grid', rr.Points3D(positions, radii=0.3, colors=[255, 100, 0]))
rr.log('world/sphere', rr.Points3D([[0, 0, 0]], radii=1.0, colors=[0, 255, 0]))
print('data logged — click on points in the viewer')

import time; time.sleep(999)
"

option B: with ground plane (click anywhere on the floor)

pip install rerun-sdk numpy

cd rerun/examples/rust/custom_callback/python_bridge

python3 -c "
import rerun as rr
import numpy as np
from ground_plane import log_ground_plane

rr.init('click_test', spawn=False)
rr.connect_grpc('rerun+http://127.0.0.1:9877/proxy')

# invisible ground plane at z=0 — click anywhere on the floor
log_ground_plane()

# some visible reference objects
grid = np.linspace(-5, 5, 20)
xx, yy = np.meshgrid(grid, grid)
positions = np.stack([xx.flatten(), yy.flatten(), np.zeros_like(xx.flatten())], axis=-1)
rr.log('world/grid', rr.Points3D(positions, radii=0.3, colors=[255, 100, 0]))
rr.log('world/sphere', rr.Points3D([[0, 0, 0]], radii=1.0, colors=[0, 255, 0]))

print('data logged — click anywhere (ground plane + entities)')
import time; time.sleep(999)
"

what to verify

• click on a colored point → terminal 1 prints world coordinates
• click on empty floor (ground plane, option B) → terminal 1 prints coordinates at z≈0
• coordinates are accurate (±0.5 of expected)
• latency feels instant (<50ms)
• multiple clicks work without drops
• viewer doesn't crash

DIM-643