Questions on Coyote testing

[pdeligia]

This is a continuation from the discussion here #399.

[jhwj9617]

Hi. Continuing from discussions here: #334
Another question - how would I know the number of test iterations to run for an automated test that satisfies a good amount of concurrency coverage. I can get a sense of degree of concurrency, # of scheduling decision, but I don't see what value of iterations would be ideal.
Obviously I could set an arbitrarily high number, but that would slow down automated testing/build processes.

[pdeligia]

Hi @jhwj9617, apologies for the delay just saw your question. This is a really good question, but there is no perfect answer. Basically, it requires a bit of trial & error as it really depends on the scenario/application you are testing. In some sense, you need to try find the sweet spot between a high enough number of iterations (to drive high coverage) versus a time budget to run these iterations (which as you say if arbitrarily high can slow down testing). When running it on your dev box (before checking in), ideally you want to be able to run at least 100 or 1000 iterations (assuming they can take say 1 min or so to run, so they don't slow down your inner loop). We have found these num of iterations working well enough in most cases we have encountered (but we have seen some very deep bugs requiring 10s of thousands of iterations to be discovered). When running on CI (or overnight), you could then run 10x (which hopefully should not take more than 10 mins assuming you run parallel CI tasks, in case of many tests). But again, these numbers really depend on how fast it is to run a single of your test iterations. If it's very fast, you can pump the iterations up, or if its slower you can lower them (but not too much else you will lose coverage).

(Btw, an open problem in our mind is how to provide more quantifiable metrics when running a coyote test, which helps avoid the above guesswork, but right now the above is the typical suggestion we provide!)

[jhwj9617]

I also noticed the Test library doesn't produce the coyote dll for net462 framework
It does show it for net5, net6 and netcoreapp3.1 though.
Is that correct, or does consumption work differently here?
@pdeligia

[pdeligia]

@jhwj9617 when you say the coyote dll for net462 framework, do you mean the coyote.dll that is basically the coyote tool executable? For the net462 framework target, it actually does not produce a dll (which you would then run via dotnet coyote.dll in .NET5+/Core), but it produces a coyote.exe instead (as it's an executable).

You can see this in the package explorer here: https://nuget.info/packages/Microsoft.Coyote.Test/1.5.4

Or maybe I misunderstood your question?

[pdeligia]

Btw, @jhwj9617 maybe for next time you could create a new top-level discussion here, just so we can organize these better & make them more discoverable for others, as your messages appeared under the "Welcome to Coyote Discussions!" topic :)

[woutervanranst]

I think I m coming across a variant of #399 (comment)

Consider this minimal example --

var expectedOrder = new List<string> { "Producer1", "Producer2", "Producer3" };
var actualOrder   = new string[expectedOrder.Count];
var currentIndex  = -1;

var t1 = Task.Run(async () =>
{
    await Task.Delay(1000);
    var index = Interlocked.Increment(ref currentIndex);
    actualOrder[index] = "Producer1";
});

var t2 = Task.Run(async () =>
{
    await Task.Delay(2000);
    var index = Interlocked.Increment(ref currentIndex);
    actualOrder[index] = "Producer2";
});

var t3 = Task.Run(async () =>
{
    await Task.Delay(3000);
    var index = Interlocked.Increment(ref currentIndex);
    actualOrder[index] = "Producer3";
});

await Task.WhenAll(t1, t2, t3);
Assert.Equal(expectedOrder, actualOrder);

This is the test output, after ~405 ms (I would expect >~3000 ms)

[coyote::test] Runtime '639d9637-7327-42cc-a1f3-405a2f99aa3d' started the test on thread '11' using the 'random' strategy.
<RandomLog> Thread '19' non-deterministically chose '3'.
<RandomLog> Thread '20' non-deterministically chose '2'.
<RandomLog> Thread '23' non-deterministically chose '2'.
[coyote::error] Unhandled exception. Xunit.Sdk.EqualException: Assert.Equal() Failure: Collections differ
                                     ↓ (pos 1)
Expected: List<string> ["Producer1", "Producer2", "Producer3"]
Actual:   string[]     ["Producer1", "Producer3", "Producer2"]
                                     ↑ (pos 1)

I would expect this test to take 3000ms or more, but then again this is the point of Coyote to control the interleaves -- but in this case the Delays are intended and I want to test the ordering -> how would I do that?