datastore: bake latest-at semantics into the garbage collector

[teh-cmc]

Consider the following log calls:

log_color("some/entity", frame_nr=0, [{255, 0, 0, 255}])
log_point("some/entity", frame_nr=1, [{1.0, 1.0}])
log_point("some/entity", frame_nr=2, [{2.0, 2.0}])
log_point("some/entity", frame_nr=3, [{3.0, 3.0}])
log_point("some/entity", frame_nr=4, [{4.0, 4.0}])
log_point("some/entity", frame_nr=5, [{5.0, 5.0}])

Querying for LatestAt("some/entity", ("frame_nr", 5)) will unsurprisingly yield a red point at (5.0, 5.0).

Now, consider what happens after running a GC that drops 50% of the data, leaving us with:

log_point("some/entity", frame_nr=3, [{3.0, 3.0}])
log_point("some/entity", frame_nr=4, [{4.0, 4.0}])
log_point("some/entity", frame_nr=5, [{5.0, 5.0}])

Querying for LatestAt("some/entity", ("frame_nr", 5)) will now yield a point at (5.0, 5.0) with whatever is currently defined as the default color, rather than red. This is just plain wrong.

This happens because the GC blindly drops data rather than doing the correct thing: compacting what gets dropped into a latest-at kind of state and keeping that around for future queries.

[emilk]

Also known as "flattening" this will be useful for our plan of storing entity properties in the store. Each edit will be added, but on save we only want the latest of every property.

[emilk]

The code is there, we just need to turn it on for the normal recordings