New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repeatable Random Tests with Fixed/Additional Seed #1461

Open
lundmikkel opened this Issue Apr 26, 2016 · 10 comments

Comments

Projects
None yet
4 participants
@lundmikkel
Member

lundmikkel commented Apr 26, 2016

(I have had a look through the documentation, some old issues and the code, but I haven't been able to find anything matching my needs, so here is my idea/suggestion)

I like random test data: it makes it difficult to (wrongly) hard-code an implementation that only matches your current test data; it removes focus from the actually used values, helping to better conveying the intend of the test; and it allows you to test the same method with new data when the seed changes, thereby possibly finding corner cases you hadn't tried in you other tests. By using NUnit's Randomizer class, random tests are even repeatable (in certain cases).

One example of a random test could be of this made-up Sort() extension method sorting integer arrays:

[Test]
public void Sort_RandomData_IsSorted() {
    // Arrange
    var random = TestContext.CurrentContext.Random;
    var data = Enumerable.Range(0, 100).Select(i => random.Next()).ToArray();

    // Act
    data.Sort();

    // Assert
    Assert.That(data, Is.Ordered);
}

From time to time, I happen to get a seed that creates random data that triggers a failing test. As long as I don't change the tests, the seed should remain the same, allowing me to rerun my test and see if I fixed the bug. Now say I want to be sure that the generated data set never breaks the method again. Then I would like to be able to rerun this test with this particular set of data, i.e. rerun the test with the seed that generated the random data when the test failed.

If I run the tests from the console, I can get the seed from the TestResult.xml file, :

<test-case
    id="0-1001"
    name="Sort_RandomData_IsSorted"
    fullname="NUnitTests.TestClass.Sort_RandomData_IsSorted"
    methodname="Sort_RandomData_IsSorted"
    classname="NUnitTests.TestClass"
    runstate="Runnable"
    seed="798259755"
    result="Passed"
    start-time="2016-04-26 13:40:09Z"
    end-time="2016-04-26 13:40:09Z"
    duration="0.023299"
    asserts="1"
/>

I would now like to be able to rerun the test with this particular seed. I imagine something like this:

[Test]
[Seed(798259755)]
public void Sort_RandomData_IsSorted() {
    // ...
}

This should allow me to set Test.Seed, so that when the test's random generator is created, that particular seed would be used.

I would, however, rather be able to set an additional seed that the test should also be run with. Something like:

[Test]
[AdditionalSeed(798259755)]
public void Sort_RandomData_IsSorted() {
    // ...
}

This should make the test run twice: once with a normal random seed, and once with the specified seed.

I am currently not aware of a way to repeat a test with a fixed seed. It seems the property on Test already has a public setter used when creating the test, but I cannot find a way to access it. Overriding it with an attribute seem to be the most elegant solution to me.

What do you think? Is this already possible? Or would it make sense to add this?

@rprouse rprouse added the is:idea label Apr 26, 2016

@rprouse rprouse added this to the Ideas milestone Apr 26, 2016

@rprouse

This comment has been minimized.

Member

rprouse commented Apr 26, 2016

This is an interesting idea.

Rather than adding two new attributes, it might be better to just have an overload of Seed that indicates that you want a run with the default behaviour. Maybe;

[Test]
[Seed]
[Seed(798259755)]
public void Sort_RandomData_IsSorted() {
    // ...
}

If we did this, I think that we should treat Seed like a TestCase. It could also have additional properties on Seed to change the behaviour, passing in a seed, using a fixed seed, default behaviour, etc. Needs a bit of design.

@lundmikkel

This comment has been minimized.

Member

lundmikkel commented Apr 26, 2016

Yeah, that could be a way to do it as well. It would also be useful to be able to require a new seed every time and that way cover more ground, so to speak. It could probably make sense to allow the attributes both on test methods and fixtures. Then you could fix the seed for all tests, if you wished.

It makes sense to do it like TestCase. That runs the test as a fixture, right?

I've previously done something similar to what I suggested above, but I simply used a method on the test class that generated a random seed and printed it, so I could late retrieve it. Now I use Randomizer, but when I work inside Visual Studio, neither the integrated test runner (using the NUnit3 Test Adapter) nor ReSharper's unit test runner allow me to retrieve the seed used for the test, so it is difficult to find the seed that triggered the failing test. Being able to print it using attributes could also be nice.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Apr 26, 2016

As mentioned in the docs, we had tossed around ideas before about how to allow this sort of repeatability at the method level. It sounds like the time is right to start firming up how we want to do it.

One thing to get out of the way first @lundmikkel - just in case the docs don't make it clear... The Seed on a test is used for two purposes:

  1. To generate the test cases themselves
  2. To generate random numbers within the test, using TestContext.Random.

Strictly speaking, the test Seed is used to generate another seed for use within the test, but it amounts to the same thing. We will need to decide which of these uses we are talking about if we put an attribute on the test. IOW, it's complicated, but undoubtedly worth figuring out.

Another related issue is how to get simple repeatability at the method level without modifying the code. That's what I had in mind when I wrote the docs page. I was thinking that nunit might pull seeds from an xml result file for those methods where it could, reverting to use of the random seed when methods were new or renamed. We should add that to the stew we are cooking here. If we decide to do it, it's probably going to be a separate issue and PR but scoping out this one may help us decide what we need in the code-based feature being discussed here.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Apr 26, 2016

@rprouse I don't think [Seed] does much to communicate what it actually does. But once we decide the functionality, I imagine a name will come to us. :-)

@rprouse rprouse added the design label Apr 26, 2016

@rprouse

This comment has been minimized.

Member

rprouse commented Apr 26, 2016

@CharliePoole you are probably right about [Seed]. It is perfectly clear within the context of this issue, but probably doesn't stand alone. It might be as simple as [RandomSeed] or a variation. Or maybe it is a property on [Test] and [TestCase].

If we are going to have a seed attribute that acts like a test case, we will also have to decide how it will behave if applied to a test with test cases. Is that an error, is it combinatorial?

I added the design label.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Apr 26, 2016

One possible idea...

  • Have a property on [TestCase] and [TestCaseSource] that controls the seed used by the randomizer that generates test cases. Let's this the TestCase generator for clarity here.
  • Continue to use the TestCase generator to create a seed for the randomizer that is exposed in TestContext, unless overridden.
  • Use Attributes on the method to override the Randomizer Seed used internally.

Where this gets tricky is if we want to run the test multiple times with different seeds. That will take a bit of thinking if the user wants to do that and also set the randomizer that generates test cases themselves.

Note that the random seed available in the xml is the one used to generate test cases plus a seed for the randomizer used within the test.

@lundmikkel

This comment has been minimized.

Member

lundmikkel commented Apr 26, 2016

@CharliePoole Okay, I was very confused about what you meant with the two different uses for the seed. I think, I figured out what you mean though: the seed is used to seed the randomizer, which is then used both when generating random test cases with parameter properties and when using the randomizer directly from the TestContext.CurrentTest.Random property, right?

If that is the case, I don't see any difference between the two uses in relation to setting the seed manually; I could just as well have a method with a Random property attribute that I wanted to control the seed of.

I like using [RandomSeed(int)] for seeding the random generator. I am, however, not sure if the attribute should be overloaded to cover all cases, as discussed above. If I read [RandomSeed(int)], I would expect the random generator to be seeded with that – and only that – seed. I would only expect it to run once.

And having a parameterless version of it, would not intuitively signal what was intended, I think.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Apr 26, 2016

I think you have it now. Think of the first randomizer as the master randomizer. It generates all the random arguments to fixture constructors and methods. This happens a long time before your tests ever run. When generating the tests, it also generates a random seed for use by your test code. The randomizer you get from the TestContext is created using that random seed.

NUnit maintains an internal dictionary of Randomizers keyed by the test method. This has no particular use in a console run since the tests are loaded and run only once. But it is used so that Gui runners - notably our own - can re-run individual tests and get the same repeated behavior.

The difference in behavior is that [RandomSeed(int)] could be used either at the time of loading the test or in running it, depending on what we want it to do.

If it replaces the seed created by the master randomizer before cases are generated, then you will get different test cases. In addition, you will get different values from TestContext.Random.

If it replaces the seed after test cases are generated, then it will only affect TestContext.Random.

The latter may be what you want since it has fewer side effects.

Another possibility for syntax is to make this an executable statement... like TestContext.Random.Seed = value.

@lundmikkel

This comment has been minimized.

Member

lundmikkel commented Apr 26, 2016

@CharliePoole Okay, I think I'm with you now 😃 I see, how setting the "outer" seed would have much bigger effect, than setting the "inner" TestContext.Random seed. And yes, I am (or that was at least my initial idea) after the inner one that changes the randomizer accessible inside my tests.

I would like the attribute approach better, as the executable statement would "pollute" the test. It is, in my eyes, meta data that we add to the test. We still want random data, but we just want to control its randomness, so to speak. That being said, I did initially try to override it in that way, but the Seed doesn't seem to be accessible from inside the test.

Setting the seed in the property would, however, have to override the existing randomizer's seed as well. Otherwise you could easily end up with differently seeded randomizers, or a randomizer that uses the old seed, even though you think it uses the new one, if it was retrieved before changing the seed. With an attribute, you set the seed before the user ever gets his hands on it, thereby avoid multiple randomizers.

@CharliePoole CharliePoole modified the milestone: Ideas Jun 24, 2016

@jnm2

This comment has been minimized.

Contributor

jnm2 commented Oct 13, 2018

The need came up at work to track down a particular seed and debug against it in Visual Studio.

Implemented [Seed(1234)]:

using System;
using System.Reflection;
using NUnit.Framework.Interfaces;
using NUnit.Framework.Internal;
using NUnit.Framework.Internal.Commands;

[AttributeUsage(AttributeTargets.Method)]
public sealed class SeedAttribute : Attribute, IWrapSetUpTearDown
{
    public SeedAttribute(int randomSeed)
    {
        RandomSeed = randomSeed;
    }

    public int RandomSeed { get; }

    public TestCommand Wrap(TestCommand command)
    {
        return new SeedCommand(command, RandomSeed);
    }

    private sealed class SeedCommand : DelegatingTestCommand
    {
        private readonly int randomSeed;

        public SeedCommand(TestCommand innerCommand, int randomSeed) : base(innerCommand)
        {
            this.randomSeed = randomSeed;
        }

        public override TestResult Execute(TestExecutionContext context)
        {
            ResetRandomSeed(context, randomSeed);
            try
            {
                return innerCommand.Execute(context);
            }
            finally
            {
                if (context.CurrentTest.Seed != randomSeed)
                    throw new InvalidOperationException($"{nameof(SeedAttribute)} cannot be used together with an attribute or test that changes the seed.");
            }
        }
    }

    private static void ResetRandomSeed(TestExecutionContext context, int seed)
    {
        context.CurrentTest.Seed = seed;

        typeof(TestExecutionContext)
            .GetField("_randomGenerator", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly)
            .SetValue(context, null);
    }
}

Sometimes you don't know the seed. You just know that it seems to fail a low percent of the time.
We also found [FindFirstFailingSeed(TimeoutMilliseconds = 1000)] helpful. It runs the test over and over, systematically increasing the random seed until the test fails. Then it prints out the seed to use to debug it.

using System;
using System.Diagnostics;
using System.Reflection;
using System.Threading;
using NUnit.Framework;
using NUnit.Framework.Interfaces;
using NUnit.Framework.Internal;
using NUnit.Framework.Internal.Commands;

[AttributeUsage(AttributeTargets.Method)]
public sealed class FindFirstFailingSeedAttribute : Attribute, IWrapSetUpTearDown
{
    public int StartingSeed { get; set; }
    public int TimeoutMilliseconds { get; set; } = Timeout.Infinite;

    public TestCommand Wrap(TestCommand command)
    {
        return new FindFirstFailingSeedCommand(command, StartingSeed, TimeoutMilliseconds);
    }

    private sealed class FindFirstFailingSeedCommand : DelegatingTestCommand
    {
        private readonly int startingSeed;
        private readonly int timeoutMilliseconds;

        public FindFirstFailingSeedCommand(TestCommand innerCommand, int startingSeed, int timeoutMilliseconds) : base(innerCommand)
        {
            this.startingSeed = startingSeed;
            this.timeoutMilliseconds = timeoutMilliseconds;
        }

        public override TestResult Execute(TestExecutionContext context)
        {
            var stopwatch = timeoutMilliseconds == Timeout.Infinite ? null : Stopwatch.StartNew();

            for (var seed = startingSeed;;)
            {
                ResetRandomSeed(context, seed);
                context.CurrentResult = context.CurrentTest.MakeTestResult();

                try
                {
                    context.CurrentResult = innerCommand.Execute(context);
                }
                catch (Exception ex)
                {
                    context.CurrentResult.RecordException(ex);
                }

                if (context.CurrentTest.Seed != seed)
                    throw new InvalidOperationException($"{nameof(FindFirstFailingSeedAttribute)} cannot be used together with an attribute or test that changes the seed.");

                if (context.CurrentResult.ResultState.Status == TestStatus.Failed)
                {
                    TestContext.WriteLine($"Random seed: {seed}");
                    break;
                }

                seed++;
                if (seed == startingSeed)
                {
                    TestContext.WriteLine("Tried every seed without producing a failure.");
                    break;
                }

                if (stopwatch != null && stopwatch.ElapsedMilliseconds > timeoutMilliseconds)
                {
                    TestContext.WriteLine($"Timed out after seeds {startingSeed}–{seed} did not produce a failure.");
                    break;
                }
            }

            return context.CurrentResult;
        }

        private static void ResetRandomSeed(TestExecutionContext context, int seed)
        {
            context.CurrentTest.Seed = seed;

            typeof(TestExecutionContext)
                .GetField("_randomGenerator", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly)
                .SetValue(context, null);
        }
    }
}

This part makes me sad though:

private static void ResetRandomSeed(TestExecutionContext context, int seed)
{
    context.CurrentTest.Seed = seed;

    typeof(TestExecutionContext)
        .GetField("_randomGenerator", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly)
        .SetValue(context, null);
}

Can we make something like ResetRandomSeed a first-class API on TestExecutionContext?
The word 'reset' is important because it starts the sequence indicated by the seed fresh from the beginning. So if you passed the current seed to this method, the seed wouldn't change but the random numbers generated would repeat from the beginning of the current seed. (A custom RepeatAttribute might want to do this to isolate an element of indeterminism besides the random generator.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment