batching 4: retire `MsgBundle` + batching support in transport layer #1679

teh-cmc · 2023-03-22T23:25:30Z

This PR finally gets rid of MsgBundle entirely, and puts DataTable in charge or serialization/deserialization.
Since DataTable support batches, the transport layer as a whole now supports batches too (which doesn't mean that the SDKs nor the store do!).

A batch always contains the following control columns:

rerun.row_id: a RowId uniquely identifying every row in the batch,
rerun.timepoint: a TimePoint for each row,
rerun.entity_path: the EntityPath that each row relates to,
rerun.num_instances: the expected number of instances for each row.

We're not yet in a position to benefit from all the niceties that batching is suppose to offer us; for that we now need to implement the changes to the DataStore on one hand and to the SDKs on the other hand.

Regressions

The new batches carry more information than the old schema (entity_path, num_instances, row_id..) and so it is expected to see some regression in very-micro benchmarks such as arrow_mono_points:

Benchmark suite	Current: `a8c0983`	Previous: `f8a263c`	Ratio
`mono_points_arrow/generate_message_bundles`	`44481992` ns/iter (`± 1123759`)	`47092378` ns/iter (`± 853262`)	`0.94`
`mono_points_arrow/generate_messages`	`178549277` ns/iter (`± 1487091`)	`123513169` ns/iter (`± 1380848`)	`1.45`
`mono_points_arrow/encode_log_msg`	`217471314` ns/iter (`± 725662`)	`157619842` ns/iter (`± 1814858`)	`1.38`
`mono_points_arrow/encode_total`	`444945548` ns/iter (`± 2328275`)	`329033469` ns/iter (`± 1723866`)	`1.35`
`mono_points_arrow/decode_log_msg`	`264431976` ns/iter (`± 1105351`)	`180134730` ns/iter (`± 875308`)	`1.47`
`mono_points_arrow/decode_message_bundles`	`92561122` ns/iter (`± 1346869`)	`53748777` ns/iter (`± 782433`)	`1.72`
`mono_points_arrow/decode_total`	`358486103` ns/iter (`± 1978345`)	`231988721` ns/iter (`± 1345301`)	`1.55`

This gets dwarfed to oblivion when you enable batching on this very same benchmark:

-    fn generate_tables() -> Vec<DataTable> {
-        (0..NUM_POINTS)
-            .map(|i| {
-                DataTable::from_rows(
-                    MsgId::ZERO,
-                    [DataRow::from_cells2(
-                        MsgId::ZERO,
-                        entity_path!("points", Index::Sequence(i as _)),
-                        [build_frame_nr(0.into())],
-                        1,
-                        (build_some_point2d(1), build_some_colors(1)),
-                    )],
-                )
-            })
-            .collect()
-    }

+    fn generate_table() -> DataTable {
+        DataTable::from_rows(
+            MsgId::ZERO,
+            (0..NUM_POINTS).map(|i| {
+                DataRow::from_cells2(
+                    MsgId::ZERO,
+                    entity_path!("points", Index::Sequence(i as _)),
+                    [build_frame_nr(0.into())],
+                    1,
+                    (build_some_point2d(1), build_some_colors(1)),
+                )
+            }),
+        )
+    }

On top of #1673
Part of #1619

Self-review
No bench regression
No test regression
No memory usage regression
api_demo_rs looking good
api_demo_py looking good
TODO all the things that fit into Tracking issue: end-to-end batching #1619
open issues for everything else

crates/re_data_store/src/log_db.rs

crates/re_log_types/benches/msg_encode_benchmark.rs

crates/re_log_types/src/arrow_msg.rs

crates/re_log_types/src/lib.rs

teh-cmc · 2023-03-25T12:53:56Z

crates/re_sdk/src/msg_sender.rs

-    /// The same component were put in the same log message multiple times.
-    /// E.g. `with_component()` was called multiple times for `Point3D`.
-    /// We don't support that yet.
-    #[error(
-        "All component collections must have exactly one row (i.e. no batching), got {0:?} instead. Perhaps with_component() was called multiple times with the same component type?"
-    )]
-    MoreThanOneRow(Vec<(ComponentName, usize)>),
-
-    /// Some components had more or less instances than some other.
-    /// For example, there were `10` positions and `8` colors.
-    #[error(
-        "All component collections must share the same number of instances (i.e. row length) \
-            for a given row, got {0:?} instead"
-    )]
-    MismatchedRowLengths(Vec<(ComponentName, u32)>),
-


These things are now checked by Data{Cell,Row,Table}

crates/re_sdk/src/msg_sender.rs

rerun_py/src/python_bridge.rs

crates/re_viewer/src/ui/event_log_view.rs

crates/re_data_store/examples/memory_usage.rs

crates/re_log_types/benches/msg_encode_benchmark.rs

crates/re_log_types/src/arrow_msg.rs