JazSharp

Notes

This repro is a fork of the original JazSharp repository. And is being maintained without huge investments.

Prerequisites

.NET 5 Runtime

Introduction

JazSharp is heavily inspired by Jasmine - the JavaScript unit testing framework. Angular developers will be very familiar with it since that is the unit testing framework used in a CLI created default app.

Why was this made?

Originally, JazSharp was only aiming to be a mocking framework but after the initial implementation attempts proved to be unreliable, the scope increased to be a full unit testing framework.

The benefits of JazSharp over other mocking frameworks are:

No reliance on interfaces or similar layers of separation that are not motivated by clean development practises.
No issues when testing code that changes static/global values for the app - statics methods and properties can be mocked just like any method.
An alternative approach to defining mocks. This is less of a benefit and more of a personal preference. For example, parameters can be checked after a mock is called and defining a mock is quick and easy.

The syntax used to define test methods is also very different from other .Net testing frameworks and allows test names (or descriptions in the case of JazSharp) to read more clearly both to technical and non-technical persons.

User Guide

Installation

To start using JazSharp, all you have to do is install the JazSharp and JazSharp.TestAdapter nuget packages from nuget.org. The former is the core of the framework and the latter enables the framework to work with Visual Studio's Test Explorer as well as allowing tests to be executed by calling dotnet test.

Creating tests

Creating the test class

A test class in JazSharp is any class that inherits from JazSharp.Spec. The class does not need to be public. Unlike the common practise in JavaScript (when using Jasmine), it is recommended to place test classes in a dedicated unit test assembly (as is popular when unit testing in .Net).

An empty test class would look something like this:

class FooSpec : Spec
{
    public FooSpec()
    {
    }
}

Describes (test scopes)

All tests need to be grouped into one or more levels of Describes. The first Describe should almost always be used to state which class is being tested.

class FooSpec : Spec
{
    public FooSpec()
    {
        Describe<Foo>(() =>
        {
        });
    }
}

Note: for static types, you will need to revert to specifying the name manually.

class StaticFooSpec : Spec
{
    public StaticFooSpec()
    {
        Describe(nameof(StaticFoo), () =>
        {
        });
    }
}

The next Describe will usually be used to specify the method being tested.

class FooSpec : Spec
{
    public FooSpec()
    {
        Describe<Foo>(() =>
        {
            Describe(nameof(Foo.Bar), () =>
            {
            });
        });
    }
}

At this point it is common to start specifying your tests but sometimes additional describes are used to group tests together by a particular scenario.

class FooSpec : Spec
{
    public FooSpec()
    {
        Describe<Foo>(() =>
        {
            Describe(nameof(Foo.Bar), () =>
            {
                Describe("when x is y", () =>
                {
                });
            });
        });
    }
}

Its (tests)

Now that you've specified the scopes for the class, method and (optionally) scenarios for your test, it is time to start defining the test itself. This is done by using the It methods. Each test's description should begin with "should" since this yields readable test descriptions.

class FooSpec : Spec
{
    public FooSpec()
    {
        Describe<Foo>(() =>
        {
            Describe(nameof(Foo.Bar), () =>
            {
                It("should initialize the flux capacitor.", () =>
                {
                    ... // test logic here
                });
            });
        });
    }
}

The above test will yield the following test description:

Foo Bar should initialize the flux capacitor.

BeforeEach and AfterEach

JazSharp allows you to specify logic that will execute before and after each test. This logic is scoped to the Describe in which it is defined. See the below code for an illustration of how the scoping works.

class FooSpec : Spec
{
    public FooSpec()
    {
        Describe<Foo>(() =>
        {
            BeforeEach(() =>        // before each 1
            {
                ...
            });

            Describe(nameof(Foo.Bar), () =>
            {
                It("should initialize the flux capacitor.", () => 
                {
                    // before each 1 will be run
                    // before each 2 will not be run
                    ... // test logic here
                });
            });

            Describe("something else", () =>
            {
                BeforeEach(() =>    // before each 2
                {
                    ...
                });
            });
        });
    }
}

Before and After Each can be defined anywhere inside the Describe - before each doesn't have to be declared before Its. It is recommended to have BeforeEach blocks at the start of a Descibe and AfterEach at the end in order to make the test code easier to follow.

Expectations (JazSharp's equivalent of Asserts - covered in a future section) can also be specified in Before and After Each blocks. This can allow you to have expectations shared among multiple tests.

Excluding and focusing

Sometimes there are tests cannot be safely re-run, are intentionally broken or are simply not relevant to pending work. In this case, it may be desirable to exclude those tests. Excluding is a simple matter of prefixing a Describe or It call with an "x". xDescribe and xIt result in a test being listed but running unit tests will result in a skipped status.

There may be other times, such as when working in a specific area or updating specific unit tests where explicity focusing those tests may be convenient. While the Test Explorer does make running specific tests easy, it is also possible to "focus" tests in JazSharp. Tests can be focused by adding an "f" prefix to a Describe or It call. If any tests are focused, only focused tests will be executed.

Custom Output

A test can add to the output log for a test by writing to the output StringBuilder. This is an optional first parameter that can be added to your test methods:

class FooSpec : Spec
{
	public FooSpec()
	{
		Describe<Foo>(() =>
		{
			It("should x y and z.", output => // output is of type StringBuilder
			{
				output.WriteLine("This is some extra test log output!");
			});
		});
	}
}

Spying

What are spies

Spies are similar to mocks except that they are applied on a per-method basis. A spy is an alternative implementation of a method which records calls made to the method and allows the test to specify alternative behaviours of the methods.

Behaviours

You can spy on a method by using the Jaz.SpyOn method. Once spied on, a method will not get executed. If the method is a function, the default value will be returned. If the method has Out parameters, these too will be defaulted.

var value = "test";
var spy = Jaz.SpyOn(value, nameof(value.ToString));
var result = value.ToString(); // result is null

You can also spy on static methods.

var spy = Jaz.SpyOn(typeof(string), nameof(string.IsNullOrEmpty));
var result = string.IsNullOrEmpty(null); // result is false

To keep the spy and execute the method's original implementation, use the CallThrough method.

var value = "test";
var spy = Jaz.SpyOn(value, nameof(value.ToString)).And.CallThrough();
var result = value.ToString(); // result is "test"

To instead return another value from the function, use the ReturnValue method.

var value = "test";
var spy = Jaz.SpyOn(value, nameof(value.ToString)).And.ReturnValue("other");
var result = value.ToString(); // result is "other"

You can also specify a sequence of return values that each subsequent call to the method will return. Once the sequence runs out, a JazSpyException will be thrown.

var value = "test";
var spy = Jaz.SpyOn(value, nameof(value.ToString)).And.ReturnValues("other", "value");
var result = value.ToString(); // result is "other"
result = value.ToString(); // result is "value"
value.ToString(); // throws exception

Alternatively, you can specify that the method should throw an exception. You can specify the exception either by passing in the instance:

var value = "test";
var exception = new InvalidOperationException();
var spy = Jaz.SpyOn(value, nameof(value.ToString)).And.Throw(exception);
value.ToString(); // throws invalid operation exception

or by specifying an exception type:

var value = "test";
var spy = Jaz.SpyOn(value, nameof(value.ToString)).And.Throw<InvalidOperationException>();
value.ToString(); // throws invalid operation exception

You can also change the parameters used for a call through by using ChangeParameterBefore one or more times.

var spy =
    Jaz.SpyOn(typeof(int), nameof(int.Parse), new[] { typeof(string) })
        .And
        .ChangeParameterBefore("s", "3")
        .CallThrough();

var result = int.Parse("5"); // result is 3

Similarly, you can set the value of an Out or Ref parameter by using ThenChangeParameter one or more times.

var spy =
    Jaz.SpyOn(typeof(int), nameof(int.Parse), new[] { typeof(string) })
        .And
        .DoNothing()
        .ThenChangeParameter("result", 9);

var result = int.Parse("5", out var parsedValue); // result is false, parsedValue is 9

DoNothing behaves the same as a default spy. The method exists solely for the purpose of calling ThenChangeParameter.

Sequences and Quantifiers

Spy behaviours can also be defined in a sequence. You got a taste for this when using ReturnValues. The following code:

var value = "test";
var spy = Jaz.SpyOn(value, nameof(value.ToString)).And.ReturnValues("a", "b", "c");

could also be written as:

var value = "test";
var spy =
    Jaz.SpyOn(value, nameof(value.ToString))
        .And.ReturnValue("a")
        .Then.ReturnValue("b")
        .Then.ReturnValue("c");

All of the spy behaviours can be specified in a sequence.

In addition to being able to specify behaviours in a sequence, you can also specify how many times each behaviour is used before moving on to the next behaviour.

The above code could also be extended out to the following code:

var value = "test";
var spy =
    Jaz.SpyOn(value, nameof(value.ToString))
        .And.ReturnValue("a").Once()
        .Then.ReturnValue("b").Once()
        .Then.ReturnValue("c").Once();

The available quantifiers are:

Once(): the behaviour executes once.
Twice(): the behaviour executes twice.
Times(x): the behaviour executes a given number of times.
Forever(): the behaviour executes forever.

Overloads

If a method has more than one overload, all overloads will be spied on. The below test demonstrates this.

class FooSpec
{
	public FooSpec() : Spec
	{
		Describe<Foo>(() =>
		{
			It("should spy on all overloads.", () =>
			{
				var spy = Jaz.SpyOn(typeof(Enumerable), nameof(Enumerable.Any));
				var test = new[] { 1, 2, 3 };

				Expect(test.Any()).ToBeFalse();
				Expect(test.Any(x => x == 2)).ToBeFalse();
				
				Expect(spy).ToHaveBeenCalledWith(test);
				Expect(spy).ToHaveBeenCalledWith(test, Jaz.InstanceOf<Func<int, bool>>());
			});
		});
	}
}

This test will pass.

Properties

In addition to spying on methods, you can also spy on properties. The following will create a spy on the Get and/or Set of methods of a property.

var array = new int[0];
var propertySpy = Jaz.SpyOnProperty(array, nameof(array.Length));

The spies for the Get and Set methods can be configured like method spies:

var array = new int[0];
var propertySpy =
    Jaz.SpyOnProperty(array, nameof(array.Length))
        .Getter.And.ReturnValue(5);
var length = array.Length; // length is 5

If the property does not have a Get method or it does not have a Set method then Getter or Setter will be null respectively.

var array = new int[0];
var propertySpy = Jaz.SpyOnProperty(array, nameof(array.Length)); // propertySpy.Setter is null

Create Spy

There are times where a spy in the form of a delegate is needed. These can be created by calling the provided Jaz.CreateSpy and Jaz.CreateSpyFunc methods. These are useful for testing events and delegates passed in as parameters.

var button = new Button();
button.Click += Jaz.CreateSpy<object, EventArgs>(out var spy);

var list = new List<int>();
list.Where(Jaz.CreateSpyFunc<int, boolean>(out var spy));

Calls

The Calls property that is available on the Spy object allows the test to retrieve information about each call to the spied on method. A case where this data is needed would be if the method being tested passes in a callback to another method. You can then get the callback from Calls[i].Arguments[j] and then test the behaviour of that callback.

Limitations

There are a few limitations on which methods can be spied on. These limitations include:

calls to base implementations of an override.

public override void ToString()
{
    return base.ToString(); // cannot be spied on
}

Calls to value type instance methods

This limitation comes from the fact that the this parameter receives special treatment for structs - it is passed by reference. This is done by the .net compiler so that methods called on a struct can change the struct's state. If this wasn't done, any changes made in a method would only affect a copy of the struct and thus changes would be lost.

This limitation may be removed in a future version of JazSharp.

var now = DateTime.Now;
now.ToString(); // cannot by spied on

Calls to possible value type instance methods

An extension to the previous limitation, if you have a generic parameter which does not have a class constraint on it, the .net compiler will pass the this parameter by reference to make sure that if the type parameter is a struct, it will execute correctly.

public void Foo<TBar>(TBar bar)
{
    bar.ToString(); // cannot be spied on
}

The workaround for this limitation is to specify the class constraint on any generic parameters you know will be class types.

public void Foo<TBar>(TBar bar)
    where TBar : class
{
    bar.ToString(); // CAN be spied on
}

Calls with more than 16 parameters

Long story short, each parameter count needs to be explicitly handled by the framework and at this time only 0 to 16 parameters are implemented. This may be extended later to support methods with more than parameters.

Executing tests in Release mode

There are outstanding issues when running JazSharp tests in a Release configuration. Only a handful of scenarios are affected by this issue and these should be resolved in a future minor version.

These are the known limitations of the current version of JazSharp. Many different kinds of method calls have been tested but other, more obscure scenarios may have been missed. The changes are low but if you encounter an InvalidProgramException or other issue then please report it.

Expectations (asserts)

JazSharp provides a sizable set of methods that can be used to check if the test was successful or not. In xUnit, NUnit and MSTest this is called an Assert. In Jasmine and JazSharp this is instead referred to as an Expect or expectation.

If an expectation fails, a JazExpectationException is thrown with a descriptive message. This exception should not be caught since it is used when detecting failed tests.

All expectations can be reversed by using Not to reverse the check. For example:

var spy = Jaz.SpyOnProperty(typeof(DateTime), nameof(DateTime.Now)).Getter;
var now = DateTime.Now;
Expect(spy).ToHaveBeenCalled(); // passes
Expect(spy).Not.ToHaveBeenCalled(); // fails

The message in the JazExpectationException is different based on whether or not the check was inverted.

Spy Expectations

One of the three supported targets for an expectation is a spy. Spies can be checked in the following ways:

ToHaveBeenCalled(): checks whether or not a spied on method was called. Does not check parameters nor call count.

var spy = Jaz.SpyOnProperty(typeof(DateTime), nameof(DateTime.Now)).Getter;
var now = DateTime.Now;
Expect(spy).ToHaveBeenCalled(); // passes

ToHaveBeenCalledTimes(x): checks whether or not a spied on method was called an expected number of times.

var spy = Jaz.SpyOnProperty(typeof(DateTime), nameof(DateTime.Now)).Getter;
var now = DateTime.Now;
now = DateTime.Now;
now = DateTime.Now;
Expect(spy).ToHaveBeenCalledTimes(3); // passes

ToHaveBeenCalledWith(...): does a deep equality comparsion checking if the given set of parameters matched any of the method calls that were made to the method.

var value = "a;b;c";
var spy = Jaz.SpyOn(value, nameof(string.Split), new[] { typeof(char[]) });
value.Split(new[] { ';' });
Expect(spy).ToHaveBeenCalledWith(new[] { ';' }); // passes

Call Expectations

A method call be wrapped in Expect in order to allow an expected exception throw to be caught. This exception can be be checked by calling ToThrow.

var exception =
    Expect(() => throw new InvalidOperationException())
        .ToThrow<InvalidOperationException>(); // passes

ToThrow will fail if the exception inherits from the expected type:

Expect(() => throw new InvalidOperationException()).ToThrow<Exception>(); // fails

Note: even if ToThrow is not called, the original exception will be caught and thus suppressed so only use a Call Expectation when checking for an expected exception.

Value Expectations

This is where the bulk of the expectation logic lies. The following checks are provided:

ToEqual(x): performs a deep equality check comparing the two given values.

var value1 = new { x = 1, y = 2 };
Expect(value1).ToEqual(new { x = 1, y = 2 }); // passes

ToBe(x): checks for an exact match between the two values.

Expect("abc").ToBe("abc"); // passes
Expect(2).ToBe(2); // passes
Expect(new object()).ToBe(new object()); // fails - different reference

ToBeTrue(): checks that the value is exactly true.

Expect(true).ToBeTrue(); // passes
Expect("true").ToBeTrue(); // fails
Expect(false).ToBeTrue(); // fails

ToBeFalse(): checks that the value is exactly false.

Expect(false).ToBeFalse(); // passes
Expect("false").ToBeFalse(); // fails
Expect(true).ToBeFalse(); // fails

ToBeDefault(): checks that the value is exactly default.

Expect(0).ToBeDefault(); // passes;
Expect(false).ToBeDefault(); // passes;
Expect((string)null).ToBeDefault(); // passes;

ToBeEmpty(): checks that the value is an empty string or enumerable.

Expect("").ToBeEmpty(); // passes
Expect(new int[0]).ToBeEmpty(); // passes

ToBeBetween(x, y): checks that the value is between x and y.

Expect(5).ToBeBetween(3, 7); // passes
Expect(new DateTime(2010, 10, 10)).ToBeBetween(new DateTime(2009, 9, 9), new DateTime(2011, 11, 11)); // passes