Large map issue with rerun/go2

[leshy]

This used to work great, yesterday we were testing go2 on roughly 500sqm or less, and my very strong machine started struggling, path wasn't showing, updates on robot position got delayed etc.

old unitree bigoffice replay is much bigger then area that has issues today

this is very little data and our computers should happily and easily process and render

---

Synced from DIM-727 by summer

[leshy]

Global mapper test on my machine

Stream("lidar"): 2263 items, 2025-12-26 11:09:06 — 2025-12-26 11:14:00 (293.9s) 
2263 frames -> 402339 voxels
  transform: 3.74s (1.7ms/frame)
  total wall: 3.75s

so 3.7 seconds for full 5 minutes of data - so why are we reducing frequency or resolution of a global mapper?

TODO - for ivan pls link above test when merged so people can confirm on their machines

My current suspicion is that rerun cannot process mesages to draw fast enough and they accumulate in some invisible buffer at the entrypoint of rerun network/drawing stack.

This is confirmed by Stashes shitty patch (reducing camera and lidar freq) helping resolve the issue
visualization/rerun/bridge.py:259

        # Rate-limit heavy data types to prevent viewer memory exhaustion.
        # High-bandwidth streams (e.g. 30fps camera, lidar) would otherwise
        # flood the viewer faster than it can evict, causing OOM.  Light
        # messages (Path, PointStamped, TF, etc.) pass through unthrottled.

This is also confirmed by a behaviour of occasinal sudden memory explosion (unrelated to explicit mem limit set in rerun bridge memory_limit (visualization/rerun/bridge.py:179) - rerun suddenly cannot keep up and more and more messages start waiting in this msg input buffer/queue

Steps:

1. confirm the theory! just send high freq pointclouds, confirm the mem explode behaviour is reproduced
2. check docs how to deal with this, ask in rerun discord channels

[ruthwikdasyam]

Problem:

viewer becomes progressively laggy. (image + lidar + global_map + costmap + odom + tf) these fall behing real-time and never recovers, delay grows continuously

DATA FLOW:

Dimos rr.log() --gRPCpipe---> gRPC proxy (1GB) ----> chunk store (7.7 GB) ---> Renderer

Cause: (Global map size increases every second)

DimOS sends data faster than viewer can ingest it
viewer must deserialize, index, store and render each message (received in Arrow-encoded format from rr.log() over gRPC)
incoming rate stays constant, data backs up in gRPC proxy, which has limit of 1GB
Once proxy fills, every msg wait in the queue, queue stays always full after
there is viewer chunk store (7.7 GB), when filled, GC drops oldest data. But, this doesnt help reduce lag

[ruthwikdasyam]

What we can do to fix it:

• publish_interval=2.5 (>0) reduces global_map update frequency, reduces how often growing payload its the pipe.
  on first look, Tested and no lag at 0.05 voxel size, at full lidar/odom/camera freq, but tehnically, expected to show lag when the map is bigger or left for long
• Can increase gRPCproxy limit from 1GB to more, but this just delays the problem/lag, doesnt solve
• Sending compressed images, reduces huge overload, but delays the lag, doesnt solve the issue (as the increase in memory is due to globalmap)

One good fix:

• dropping msgs at rr.log().
  when 1 gb of gRPC is full, it puts backpressure and rr.log() could block for long time waiting to push into the pipe, so approach to dropping msgs, means skipping rr.log() call for stale msgs works (condition to drop, if msg.ts is more than 500ms ago)
  thus, never blocks.

• This, along with one or more of above fixes, should be good

[leshy]

publish_interval=2.5 (>0) reduces global_map update frequency, reduces how often growing payload its the pipe.

size of global map pointcloud is not huge. we will hit the same limit elsewhere soon, I know we can drop the data on our end but this heavily depends on machine used, and we shouldn't nanny rerun like this, rerun itself should know when to drop, can we control this on rerun end? can we measure this internal buffer of rerun to know when and how often to drop?

if we do drop we won't do this by modifying our global mapper rate but rerun bridge, if unsolvable on rerun side we will spec this out (we shouldn't modify the thing being visualized if visualizer struggles, the thing being visualized is fine, visualizer has a problem)

[leshy]

Usually in these types of systems you can control backpressure strategies per-stream, we should allow rerun to drop pointcloud or image messages if renderer is struggling basically

[ruthwikdasyam]

The heavy global map goes through snapping+unique in the Rerun callback, even though these points are already unique and snapped

In VoxelGridMapper, when the pointclouds are processed, a hashmap is used and points are inherently snapped. But in the Rerun callback, when it receives LCM global map messages, it goes through np.floor and np.unique before calling rr.log() (publishing to Rerun). This is causing the lag - each global map message is taking up to 250ms (160k voxels), and this is blocking other lidar/image messages along with creating a buffer on the LCM side.
So the lag you see on Rerun is because messages are being sent late to Rerun, and Rerun is just doing its job.

After the 2nd redundant snapping is skipped, the globalmap processing time reduced to 40ms on avg (0.05 voxel size, 250k voxels).