-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kvs: add eventual consistency test coverage #1832
Comments
Replying to a comment from @garlick in #1863.
My initial thought was creating some type of pause / unpause of all setroot events. When setroots are paused, queue them up, then send them out on an unpause. So imagine we can do tests like this: flux kvs put a=0 i believe the above test effectively tests read-your-writes (maybe they should be flux kvs gets instead, but the main idea is the same). A variant should be able to test causal I think. Dunno about monotonic, I think that might be implicitly taken care of with many of the stress / racy tests? |
When I initially thought about this, I thought about pausing the "send" of setroot events. It ends up this won't work. I had forgotten that setroot events also contain the fence names used during a transaction. If a rank can't receive those names, then transactions will never complete. What may work is actually pause / unpause of "received" setroot events. So the test would be a bit different. flux kvs put a=0 |
Oh, because the This approach seems equivalent so sounds good to me! |
I think you have the right idea, but not the initial "a=0" line. In my original example far above, the:
Would have hung. With the new approach, hangs can still happen. So the tests just have to be careful which ranks they pause and which ones they don't. |
shall we consider this issue closed? #1907 added read-your-writes. I believe monotonic consistency is handled via normal tests and causal is handled via all of the "wait for version" tests Or perhaps all that has to be done is to document that those tests test these conditions? |
I think we're good for now. Thanks! |
The kvs makes several guarantees for eventual consistency:
We should have tests for these specific guarantees.
Since the relaxed consistency is mainly due to the open loop setroot event, we might consider adding test hooks that allow the setroot event publishing to be delayed until some triggering event, so that non-racy tests can be written to validate these guarantees.
The text was updated successfully, but these errors were encountered: