Hosted data collection / storage

[spomichter]

Dimenisonal and server-side infrastructure for hosted data collection for developers

Requirements

• Integrates with memory2 (current local storage system)
• Integrate with current replay system so i can can replay from a server
• Auto detection of US versus China servers
• Auto detection of network environment / dynamically propose most optimal upload options (every hour versus every day etc)
• Support for both realtime data collection storage and asynchronous data storage
• CLI tool for uploading

Use cases:

• Friend uploads his replay in one place, i can replay right after he uploads in another place, super great for sharing issues, bugs, etc with the dimensional
• Cloud storage for developers if their dataset is huge

[xiaoyaoqilan]

The first implementation stack for this proposal is now available as six draft PRs:

1. Add WebRTC cross-region and replay repository MVP #3177 — WebRTC cross-region latency + foundational replay repository MVP
2. Add S3-compatible replay object storage #3184 — S3-compatible object storage (S3/R2/MinIO)
3. Add indexed memory2 replay uploads #3185 — WAL-consistent memory2 snapshot upload and dataset index
4. Resolve remote memory2 replays for Go2 #3186 — remote replay resolution for dimos --replay run unitree-go2
5. Select healthy China and US replay nodes #3187 — concurrent China/US node probing with a one-hour recommendation cache
6. Harden replay repository operations #3188 — TLS, quotas, metrics, CDN URLs, audit logging, and interrupted-upload recovery

These PRs are intentionally stacked and kept as drafts so each layer can be reviewed independently. The final stacked branch passes the 56-test targeted suite, Ruff, formatting, and targeted mypy checks.

A real H.264 MP4 was also uploaded to the Alibaba Cloud repository service and downloaded/hash-verified on the server. Public port access still requires the Alibaba Cloud security-group entry for the selected service port; that deployment setting is separate from the repository implementation.