feature request: add a loose upper bound for skew amongst shards in VStream based on timestamps

[gordonyoon]

Feature Description

VStream currently has no coordination amongst shards for ordering. Although we do not want a strict ordering of messages across shards, we would like some light coordination (something like an upper bound of 1 second) such that transactions executed at approximately the "same time" are sent closely together in VStream.

Use Case(s)

We have a query-cache system that ingests changes from VStream. When the query-cache receives updates, it sends the updates to its clients, which in turn will update their UIs. Our current vschema has tables sharded by design files. Although the files can be mostly independent, it would be strange behavior if the time skew amongst shards was completely unbounded.

For example (this is a contrived case for illustration), if an image is moved from one file to another. That change does not have to be executed atomically for clients; however, if the changes come in with a 5 minute difference, it would be strange to see the image disappear from file1, and appear in file2 after 5 minutes.

[rohit-nayak-ps]

@gordonyoon, VStream currently streams events as they arrive from any shard in the list of shards provided in the vgtid. So unless there is a very large lag on the replica of one of the shards, we should not see events arriving in near-realtime and thus events across shards should come closely together. Can you provide a situation where you have actually experienced this?

[rohit-nayak-ps]

Closed via #7626