Patrick Lioi edited this page Sep 3, 2018 · 61 revisions

Quick Start

Fixie is a .NET test framework similar to NUnit and xUnit, but with an emphasis on low-ceremony defaults and flexible customization.

Add a test project to your solution. Target net452 or higher, netcoreapp2.0 or higher, or a combination. Reference the Fixie and Fixie.Console NuGet packages. Install a third-party assertion library such as Shouldly. Unlike MSTest and xUnit, do not reference Microsoft.NET.Test.Sdk.

<!-- Example.Tests.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>netcoreapp2.0;net471</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Fixie" Version="2.0.2" />
    <PackageReference Include="Shouldly" Version="2.8.3" />
    <DotNetCliToolReference Include="Fixie.Console" Version="2.0.2" />
  </ItemGroup>

</Project>

Add a test class to the project. By default, a test class is any class whose name ends with Tests, and public methods in that class will be treated as tests. The test class will be instantiated once per test method, like xUnit does. These defaults can be customized, but provide a simple starting point:

using Shouldly;

public class CalculatorTests
{
    public void ShouldAdd()
    {
        var calculator = new Calculator();
        calculator.Add(2, 3).ShouldBe(5);
    }

    public void ShouldSubtract()
    {
        var calculator = new Calculator();
        calculator.Subtract(5, 3).ShouldBe(2);
    }
}

To run your tests from the command line, navigate into the test project folder and execute dotnet fixie:

$ cd Example.Tests
$ dotnet fixie

Running Example.Tests (netcoreapp2.0)

2 passed, took 0.02 seconds

Running Example.Tests (net471)

2 passed, took 0.01 seconds

To run your tests from inside Visual Studio, open Test Explorer (Test \ Windows \ Test Explorer). Rebuild your solution and click Run All to get started.

Customizing the Test Assembly Lifecycle

.NET test frameworks have two phases: discovery and execution.

In the discovery phase, the framework inspects your test project and finds all the test classes and test methods. For instance, NUnit scans for attributes like [Test], xUnit scans for [Fact], and Fixie scans for the test class naming convention seen above. The discovery phase may happen all on its own, such as when Visual Studio needs to find out the full list of available tests to display within Test Explorer.

In the execution phase, the discovered test classes are instantiated, the discovered test methods are invoked, and their results are tallied and reported.

Each discovered test method name is a Test. When such a test method is parameterized, that Test may be run multiple times. Each specific run of a Test is called a Case, and each Case can individually pass, fail, or be skipped.

Customizing the Discovery Phase

To customize the test discovery phase, place a subclass of Discovery in your test project. From the constructor, you can customize the discovery phase's behavior for discovering test classes, test methods in those classes, and input parameters for parameterized test methods.

For example, say you wish to mark test classes with a [TestClass] attribute instead of the default "*Tests" naming convention, and you wish to mark test methods with a [Test] attribute instead of the default inclusion of all public methods. Additionally, you'd like to allow for parameterized tests by placing [Input(1, "B", 3.0)] attributes on a test method. Begin by defining those attributes:

[AttributeUsage(AttributeTargets.Class)]
public class TestClass : Attribute { }

[AttributeUsage(AttributeTargets.Method)]
public class Test : Attribute { }

[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class Input : Attribute
{
    public Input(params object[] parameters)
    {
        Parameters = parameters;
    }

    public object[] Parameters { get; }
}

Next, customize the discovery phase:

using Fixie;

public class TestingConvention : Discovery
{
    public TestingConvention()
    {
        Classes
            .Where(x => x.Has<TestClass>());

        Methods
            .Where(x => x.Has<Test>());

        Parameters
            .Add<InputParameterSource>();
}

public class InputParameterSource : ParameterSource
{
    public IEnumerable<object[]> GetParameters(MethodInfo method)
    {
        var inputAttributes = method.GetCustomAttributes<Input>().ToArray();

        foreach (var input in inputAttributes)
            yield return input.Parameters;
    }
}

Lastly, write test classes which use your new testing convention:

[TestClass]
public class CalculatorTester
{
    public void NotATest()
    {
        //This is no longer treated as a test, because it lacks the [Test] attribute.
    }

    [Test]
    public void ShouldAdd()
    {
        var calculator = new Calculator();
        calculator.Add(2, 3).ShouldBe(5);
    }

    [Test]
    [Input(5, 3, 2)]
    [Input(8, 5, 30)]
    public void ShouldSubtract(int x, int y, int expectedDifference)
    {
        //This Test is invoked twice, once for each [Input(...)],
        //resulting in two Cases, the first of which passes and the
        //second of which fails.
        var calculator = new Calculator();
        calculator.Subtract(x, y).ShouldBe(expectedDifferece);
    }
}

Customizing the Execution Phase

By default, the test class execution phase performs the following steps for each discovered test class:

For each test method in the test class:
    Construct the test class using its default constructor.
    Invoke the test method.
    Dispose of the test class instance, if it is IDisposable.

In other words, your test classes are constructed frequently, like xUnit, meaning that you can use the constructor and Dispose() methods for test setup and test teardown.

However, if you don't want to use this default test class lifecycle, you can define your own. For example, say you wish to instead have the infrequent test class construction familiar to NUnit users:

Construct the test class using its default constructor.
For each test method in the test class:
    Invoke the test method.
Dispose of the test class instance, if it is IDisposable.

To customize the test execution phase, place a subclass of Execution in your test project.

using Fixie;

public class TestingConvention : Execution
{
    public void Execute(TestClass testClass)
    {
        var instance = testClass.Construct();

        testClass.RunCases(@case =>
        {
            @case.Execute(instance);
        });

        instance.Dispose();
    }
}

Lastly, write test classes which use your new testing convention:

public class CalculatorTests
{
    private readonly Calculator calculator;

    public CalculatorTests()
    {
        calculator = new Calculator();
    }

    public void ShouldAdd()
    {
        calculator.Add(2, 3).ShouldBe(5);
    }

    public void ShouldSubtract()
    {
        calculator.Subtract(5, 3).ShouldBe(2);
    }
}

With this testing convention in place, CalculatorTests would be constructed once, and the Calculator instance would be shared across both tests.

For completeness, this means that the default Execution implementation, when you provide no such Execution class yourself, is:

class DefaultExecution : Execution
{
    public void Execute(TestClass testClass)
    {
        testClass.RunCases(@case =>
        {
            var instance = testClass.Construct();

            @case.Execute(instance);

            instance.Dispose();
        });
    }
}

By defining your own Execution, you control every aspect of the test class lifecycle: how to construct the test class (you might have some opinion about the meaning of test class constructor arguments), construction frequency, what steps to take before and after each test class, and what steps to take before and after each test case.

Putting it all Together

Imagine you start testing a new system and you occasionally find that the test methods in your test class all share a few setup steps. You might factor that code into the test class constructor, allowing each test method to focus on the individual scenario at hand.

That's great, until you begin to test a feature that exposes async methods. Test methods can be async, but C# constructors cannot. It might seem that you're stuck with leaving out the test class constructor, and repeating the async setup steps within each test method.

Instead, we can take over both the discovery and execution phases in order to describe a synchronous-or-asynchronous test setup convention:

using System;
using System.Linq;
using System.Reflection;
using Fixie;

public class TestingConvention : Discovery, Execution
{
    public TestingConvention()
    {
        Methods
            .Where(x => x.Name != "SetUp");
    }

    public void Execute(TestClass testClass)
    {
        testClass.RunCases(@case =>
        {
            var instance = testClass.Construct();

            SetUp(instance);

            @case.Execute(instance);
        });
    }

    static void SetUp(object instance)
    {
        instance.GetType().GetMethod("SetUp")?.Execute(instance);
    }
}

Here, we customize the discovery phase so that if a method appears in a test class but is named "SetUp", Fixie will not consider it to be a test method. SetUp(), if present, will no longer be invoked by the test runner during the test class lifecycle.

Next, we customize the execution phase so that we will invoke SetUp() ourselves, exactly when we want to: right after test class instantiation, right before a test case is executed.

The final call to the MethodInfo.Execute(object instance, params object[] arguments) helper method works whether or not a test class's optional SetUp() method is async, ensuring the task is properly awaited.

With this convention, we can write test classes in a few ways:

  1. A test class could have no SetUp() method and will run just like the default behavior.
  2. A test class could have a synchronous public void SetUp(), and it will be invoked right before each test case.
  3. A test class could have an asynchronous public async Task SetUp(), and it will be invoked and awaited right before each test case.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.