Suggestion for reproducing test sequence for a random case in your "Fuzzy Tests"

[tohagan]

Thanks so much for the hard work you put into the latest June release.
I was particularly delighted to see the attachment performance improvements.

I had a look at your discussion on the new Fuzzy Tests and thought you might want to consider replacing Math.random() with a random number generator that you can "seed" in order to be able to reproduce a randomly generated test sequence.

Here's how I imagined you might do this.

1. var seed = Math.random(); and write it to your test log.
2. replace Math.random() with a seedable version (see link below)
3. Seed the new Math.random() function using "seed" variable
4. Now run your existing Fuzzy Test suite using the new Math.random()

Now if you need to rerun the same test sequence, just set seed to the value you logged instead of a random value (replace Step 1) and this should generate the same test sequence.

https://github.com/davidbau/seedrandom

[tohagan]

Once you find a seeded test case that fails, you can even add it permanently to your tests by calling the Fuzzy Test suite with the known seed value that causes the failure and treating this call as a new unit test.

[tohagan]

Just realised that one issue might bite you ...

If PouchDB uses Math.random() (say for random backoff when retrying) and the number of calls to this method was dependent on external random events (like network response times) then this might wreck the predictability of the pseudo random sequence used to generate the tests. The same lack of predictability might occur between browsers if say event firing order was different. Even if this was not an issue today it might be in the future as the code in PouchDB evolves.

So I think you might wiser to just use one of the seeded PRNG functions available in the lib I gave you within your Fuzzy Unit test code instead of calling Math.random(). This will give you a separate random sequence to that used within the PouchDb lib under test. You might still want to seed and replace Math.random() to improve the predictability of it's use within the PouchDb lib - but keep this random sequence separate from the one you use to generate the tests.

Unclear? I'm happy to catch up for a chat or Skype call.

[daleharvey]

I have a few ideas about how to implement this and given a cursory look over this idea it sounds by far the best, we use math.random for uuid's and backoff in pouchdb itself, if they used the seeded version also that would be fine.

Ill take a shot at implementing this, thanks so much for the idea, seriously if it works out its 100x less work than I expected

[tohagan]

What I think you finally want to be able to do is to run a predicable stress + regression test. This might be a new test suite or an adaptation of your existing tests (I've not had a good look over what tests you've already done). I've used the PRNG trick for generating this kind of testing.

To clarify ... By "stress testing" I mean that you want to be able to run a stressfully large number of PouchDb clients against a single CouchDb server over a prolonged time period and apply regression assertions to each change that occurs to each PouchDb client during Push or Pull replication. You'd also want to perform a regression assertion on the final state of each PouchDb and CouchDb server once the prolonged stress session has completed. You might even want to try running a single client with multiple CouchDb database instances replicating concurrently.

You want to set up the test suite to use variables that configure stuff like: number of clients, direction of replication, replication type, persistence type (WebSQL/IndexDb), browser type etc so the test suite is fully parameterized. Then you can run batch tests using combinations of variables.

To further stress the system, you can use a proxy to simulate random network stops/starts/packet loss/latency. Of course you'll want to log everything so you can identify causes. I recommend doing this in a data format that can be filtered/sorted and be able to turn off logging in the event that it slows the code down too much and thus prevents triggering some test case. Include timing in your logging as you may also want to regression expected response time (with a margin of error).

True storey .. I once had to deliver a public transport journey planner that was to be widely publicised by an Australian state government. It had to work perfectly under stress from day one. My boss says ... "Tony is it ready yet?" ... I replied "Nahhh .. gimme another week". That week I wrote a stress/regression test suite and found two critical bugs in this complex multi-threaded C++ app. First day of use we had 8,000 app visitors. That week saved us! We later won two IT awards for this app in two different states in Australia and it eventually was sold to other countries. It's still in use today.

[tohagan]

You can use Wireshark to perform the network logging and also to be the proxy that simulates network stops/starts/packet loss/latency. Works with SSL and can reassemble packet streams & socket sessions.

[NickColley]

@tohagan love the simplicity of your idea and also your explanation, thanks for sharing 👍 :)

[kxepal]

@tohagan another idea is to record failure cases into database which is easy to share and use it as source to reproduce the fail cases on the other dev machine. Since PouchDB have a power to sync with CouchDB, this should be trivial to maintain and distribute such database across all the devs.

[tohagan]

Another potential gotcha with random seed generated tests is that on different CPU types the random sequence might not match. This less likely to be an issue if the PRNG algorithm does not use floating point numbers. In practice this may not be an issue since we have better standardisation of encoding and computation of floating point numbers but worth being aware of.

[daleharvey]

@kxepal I have done a similiar thing to that in the past and what I was preparing to do when I opened this thread, it however is fairly complicated, coordinating the logs over multiple pouchdb instances is messy plus you need to define a log format, how so save it / share it and load it into your tests.

I think it would be a super useful thing to add to pouch (plugins likely) but for now I am going to go for the seedrandom idea as it looks super easy to implement, to replay the logs people just need the seed

[daleharvey]

Forgot to close as I landed this, cheers