Validate etcd linearizability #14398
Conversation
serathius
force-pushed
the
linearizability
branch
5 times, most recently
from
c5a30cb
to
7615605
Compare
serathius
force-pushed
the
linearizability
branch
from
7615605
to
7bd9685
Compare
serathius
force-pushed
the
linearizability
branch
3 times, most recently
from
16d36e0
to
320ef73
Compare
serathius
added
the
priority/important-soon
serathius
force-pushed
the
linearizability
branch
from
320ef73
to
e30f0e2
Compare
Codecov Report
@@ Coverage Diff @@
## main #14398 +/- ##
==========================================
- Coverage 75.70% 75.44% -0.27%
==========================================
Files 457 457
Lines 37300 37269 -31
==========================================
- Hits 28239 28116 -123
- Misses 7309 7380 +71
- Partials 1752 1773 +21
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
serathius
force-pushed
the
linearizability
branch
4 times, most recently
from
ac43a74
to
697a691
Compare
serathius
force-pushed
the
linearizability
branch
9 times, most recently
from
732a65c
to
1894704
Compare
serathius
changed the title
|
@ahrtr @spzala I think this is ready to review. This PR provides first scenario to new framework of linerazibility test. Using this approach I was able to reproduce both data durability and data inconsistency issues. Further work should add more diverse scenarios to cover broader types of failures and allow dynamic scenarios based on random set of actions. Generating random scenarios should allow us to cover much broader space and should be our main goal. Current approach based on prepacked scenarios is limited to testing historical or simple scenarios. By testing a generic system property like linearizability we are no longer limited by simple Next steps:
|
serathius
force-pushed
the
linearizability
branch
from
1894704
to
5779393
Compare
serathius
force-pushed
the
linearizability
branch
11 times, most recently
from
cdc5957
to
09909f5
Compare
serathius
force-pushed
the
linearizability
branch
5 times, most recently
from
0ce584a
to
b26705d
Compare
There was a problem hiding this comment.
@serathius great work! My only concern is that thegithub.com/anishathalye/porcupine seems developed/maintained by an individual, and there is not much to find out on dev history/PRs. But at the same time, we don't have other options and also that we are using it for test. So giving porcupine a try sounds okay. (Forking and maintaining our own copy may be too much of work specially when we don't have enough contributors so I guess we don't have that option.) I don't have any other comment besides I noticed a question from @ahrtr Thanks!
| ) | ||
|
|
||
| var ( | ||
| PutGetTraffic Traffic = putGetTraffic{} |
There was a problem hiding this comment.
It doesn't make sense to define a global variable. We should either define a function something like NewTraffic or users use putGetTraffic{} directly.
There was a problem hiding this comment.
I want to maintain a single clean place to that lists are available traffic and failpoints. I find it more clear than using structs and asking user to scan whole file where they are scattered.
| }, | ||
| { | ||
| name: "KillClusterOfSize3", | ||
| failpoint: KillFailpoint, |
There was a problem hiding this comment.
| failpoint: KillFailpoint, | |
| failpoint: killFailpoint{}, |
| traffic := trafficConfig{ | ||
| minimalQPS: minimalQPS, | ||
| maximalQPS: maximalQPS, | ||
| clientCount: 8, |
There was a problem hiding this comment.
minor comment: please consider to make this configurable, such as adding a field clientCount into the struct tcs.
There was a problem hiding this comment.
Struct tcs is not set in stone, we can always add it when we decide to parameterize tests based on it.
|
Overall looks good to me with some minor comments. Great work!
Yes, I have the same concern, especially if we want to spend more resource/time on the linearizability test. We can discuss this separately. |
Signed-off-by: Marek Siarkowicz <siarkowicz@google.com>
serathius
force-pushed
the
linearizability
branch
from
b26705d
to
069e26e
Compare
First draft of linearizability tests that are able to reproduce #14370 within 20 seconds with 80% accuracy. Part of #14045
This approach uses a generic way to of verifing linearizability. In this proof of concept to reproduce #14370, however for full solution scenarios should be generated randomly based on preexisting fail points.