Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request: add a loose upper bound for skew amongst shards in VStream based on timestamps #7402

Closed
gordonyoon opened this issue Jan 27, 2021 · 2 comments

Comments

@gordonyoon
Copy link

Feature Description

VStream currently has no coordination amongst shards for ordering. Although we do not want a strict ordering of messages across shards, we would like some light coordination (something like an upper bound of 1 second) such that transactions executed at approximately the "same time" are sent closely together in VStream.

Use Case(s)

We have a query-cache system that ingests changes from VStream. When the query-cache receives updates, it sends the updates to its clients, which in turn will update their UIs. Our current vschema has tables sharded by design files. Although the files can be mostly independent, it would be strange behavior if the time skew amongst shards was completely unbounded.

For example (this is a contrived case for illustration), if an image is moved from one file to another. That change does not have to be executed atomically for clients; however, if the changes come in with a 5 minute difference, it would be strange to see the image disappear from file1, and appear in file2 after 5 minutes.

@rohit-nayak-ps
Copy link
Contributor

@gordonyoon, VStream currently streams events as they arrive from any shard in the list of shards provided in the vgtid. So unless there is a very large lag on the replica of one of the shards, we should not see events arriving in near-realtime and thus events across shards should come closely together. Can you provide a situation where you have actually experienced this?

@rohit-nayak-ps
Copy link
Contributor

Closed via #7626

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants