tests: design for a network simulator debugging tool

[andybons]

Megastore has a pseudo-random test framework capable of exploring the space of all possible orderings and delays of communications between simulated nodes or threads, and deterministically reproduces the same behavior given the same seed.

This would be unbelievably useful.

/cc @toddlipcon since he’s had experience designing something like this for HDFS HA.

[andybons]

https://github.com/apache/hadoop-common/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQJMWithFaults.java#L202

[strangemonad]

have a look at Jepsen:
https://github.com/aphyr/jepsen

Kyle’s done a great job with simulating failures and network partitions for many DBs

shawn

On Jun 4, 2014, at 4:48 PM, Andrew Bonventre notifications@github.com wrote:

https://github.com/apache/hadoop-common/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQJMWithFaults.java#L202

—
Reply to this email directly or view it on GitHub.

[strangemonad]

have a look at Jepsen:
https://github.com/aphyr/jepsen

and the series of blog posts on breaking serializability in distributed DBs:
http://aphyr.com/tags/jepsen

[andybons]

Sweet.

On Wed, Jun 4, 2014 at 5:28 PM, Shawn Morel notifications@github.com
wrote:

have a look at Jepsen:
https://github.com/aphyr/jepsen

and the series of blog posts on breaking serializability in distributed
DBs:
http://aphyr.com/tags/jepsen

—
Reply to this email directly or view it on GitHub
#27 (comment)
.

[spencerkimball]

Wow, Jepsen seems great. We should obviously use it. Thanks Shawn!

On Wed, Jun 4, 2014 at 5:29 PM, Andrew Bonventre notifications@github.com
wrote:

Sweet.

On Wed, Jun 4, 2014 at 5:28 PM, Shawn Morel notifications@github.com
wrote:

have a look at Jepsen:
https://github.com/aphyr/jepsen

and the series of blog posts on breaking serializability in distributed
DBs:
http://aphyr.com/tags/jepsen

—
Reply to this email directly or view it on GitHub
<
https://github.com/cockroachdb/cockroach/issues/27#issuecomment-45154135>
.

—
Reply to this email directly or view it on GitHub
#27 (comment)
.

[jonasschneider]

For randomized testing, instead of the "attack-based" approach of Jepsen, there's also this: https://github.com/stripe-ctf/octopus -- @zenazn might be interested.

[spencerkimball]

Cool, octopus sounds helpful.

On Fri, Aug 15, 2014 at 8:31 AM, Jonas Schneider notifications@github.com
wrote:

For randomized testing, instead of the "attack-based" approach of Jepsen,
there's also this: https://github.com/stripe-ctf/octopus -- @zenazn
https://github.com/zenazn might be interested.

—
Reply to this email directly or view it on GitHub
#27 (comment)
.

[xiang90]

We are exploring something similar for etcd.
Personally, I do not think jespen is good enough for this kind of test.

[xiang90]

@andybons @spencerkimball We just started a pseudo-random test framework inside raft.

The first step is to randomly probing the possible testing space with a seed, so that we can reproduce the found problem.
The second step is to actually recording the probed testing space and find a way to end the testing, when we are sure that we have searched all possible sequences for the internal state machine.

etcd-io/etcd#2203

[andybons]

@xiang90 👍

[bdarnell]

Making a note of this for later: https://github.com/ahorn/linearizability-checker is another tool for analyzing Jepsen logs. It's similar to but more efficient than Aphyr's Knossos tool.

[knz]

Ok this is really a superset of what I was looking at with #4036. Let me study these options and then I'll see what I can propose.

[AkihiroSuda]

We have also a similar tool which increases the non-deternimism of thread scheduling (and so on).
This tool can be easily used with the existing unit tests and Jepsen tests.

http://osrg.github.io/earthquake/post/fosdem2016/

[knz]

I'm not sure this is happening in this way any more. Not planning to work on this soon.

[knz]

cc @knz for later

[bdarnell]

We've been using jepsen for a long time now and have no specific plans to build anything else in this area, so there's no point in keeping this issue open.