Test generators allow tracks to generate tests automatically without having to write them ourselves. Each test generator reads from the exercise's canonical data, which defines the name of the test, its inputs, and outputs. You can read more about exercism's approach to test suites here.
Generating tests automatically removes any sort of user error when creating tests. We want the tests to be accurate with respect to its canonical data. Test generation also makes it much easier to keep tests up to date. As the canonical data changes, the tests will be automatically updated when the generator for that test is run.
An example of a canonical data file can be found here
When looking through the canonical data and the generator code base, we use a lot of common terminology. This list hopefully clarifies what they represent.
- Canonical Data - Represents the entire test suite.
- Canonical Data Case - A representation of a single test case.
- Description - The name of the test.
- Property - The method to be called when running the test.
- Input - The input for the test case.
- Expected - The expected value when running the test case.
Adding a test generator is straightforward. Simply add a class to the Generators.fs file in the generators folder with the name of the exercise (in PascalCase), and extend the GeneratorExercise abstract class.
An example of a simple generator would be the Bob exercise. The source code can be found below, but you can also view it in the repository here.
type Bob() =
inherit GeneratorExercise()This is a fully working generator, no other code needs to be written. However, it's simplicity stems from the fact that the test suite and the program itself are relatively trivial.
For many generators, the default behavior of the GeneratorExercise class suffices. However, for more complex generators, one can override one or more of its virtual methods:
Render the input of a test method.
The Gigasecond generator renders the input value as a (parenthesized) DateTime value:
override __.RenderInput (_, _, value) =
DateTime.Parse(string value, CultureInfo.InvariantCulture)
|> DateTime.renderParenthesizedNote that you could use the key field to only customize the output for specific input fields. The Yacht generator does this:
override __.RenderInput (canonicalDataCase, key, value) =
match key with
| "category" -> Obj.renderEnum "Category" value
| _ -> base.RenderInput (canonicalDataCase, key, value)Render the expected value of a test method.
The Connect generator renders the expected value as an Option<T> value:
override __.RenderExpected (canonicalDataCase, key, value) =
match string value with
| "O" -> "(Some White)"
| "X" -> "(Some Black)"
| _ -> "None"Render the System Under Test (SUT). This is the actual value that is verified against the expected value.
The RunLengthEncoding generator has a test case where it verifies that encoding and then decoding the input results in the same value. The RenderSut method is overridden to convert what would be a single call to a non-existing consistency function, to a chain of encode |> decode calls:
override this.RenderSut canonicalDataCase =
match canonicalDataCase.Property with
| "consistency" ->
let parameters = this.RenderSutParameters canonicalDataCase |> String.concat " "
sprintf "%s |> encode |> decode" parameters
| _ ->
base.RenderSut canonicalDataCaseRender the arrange part of the test method. In this part, the SUT is created.
The default behavior is usually what you want. Only in very special circumstances would you want to change this.
Note: this method returns a string list, where each string corresponds to a single, rendered line in the test file.
Render the assert part of the test method. In this part, the assertion is made to verify the SUT against the expected value.
The default behavior is usually what you want. Only in very special circumstances would you want to change this.
Note: this method returns a string list, where each string corresponds to a single, rendered line in the test file.
Render any additional utility/setup methods. This can be used to define helper methods that can be used in the test methods.
The setup code is added before the test methods, to enable the setup methods to be used in the test methods.
Note: this method returns a string list, where each string corresponds to a single, rendered line in the test file.
The Zipper generator adds two helper methods that are used in the test methods to remove some tedious, boilerplate code:
override __.RenderSetup _ =
[ "let subTree value left right = Some (tree value left right)"
"let leaf value = subTree value None None" ]
|> String.concat "\n"Render a single value (which is used to render input parameters, the expected value and the SUT).
The ComplexNumbers generator renders different values based on the value's type:
override __.PropertiesWithIdentifier canonicalDataCase =
match canonicalDataCase.Expected.Type with
| JTokenType.Array -> ["sut"]
| _ -> base.PropertiesWithIdentifier canonicalDataCaseMap the canonical data case. This can be useful if you want to add or remove properties.
The default behavior is usually what you want. Only in very special circumstances would you want to change this.
The ScaleGenerator generator uses the canonical data case mapping to add a missing property as a null value (which makes processing later easier):
override __.MapCanonicalDataCase canonicalDataCase =
let input = canonicalDataCase.Input
match Map.tryFind "intervals" input with
| Some _ -> canonicalDataCase
| None -> { canonicalDataCase with Input = Map.add "intervals" null input }Specifies which properties should be used as parameters to the SUT.
The default behavior is usually what you want. Only in very special circumstances would you want to change this.
The RailFenceCipher explicitly defines the properties used as SUT parameters to change the order in which they are passed:
override __.PropertiesUsedAsSutParameter _ = ["rails"; "msg"]Specifies which properties should have an identifier assigned to them. If a property is in this list, it will be assigned its value on a separate line, to its own identifier.
The Transpose generator has chosen to use identifiers for all properties:
override this.PropertiesWithIdentifier canonicalDataCase = this.Properties canonicalDataCaseIn some cases, you want an identifier to have an explicit type. A common use case is when a value is an empty list. In those case, FsUnit needs a type annotation for it to successfully execute the assertion.
The Minesweeper generator adds a string list type annotation for empty values:
override __.IdentifierTypeAnnotation (_, _, value) =
match Seq.isEmpty value with
| true -> Some "string list"
| false -> NoneReturns a list of additional namespaces to open in the test file.
The Gigasecond generator works with DateTime instances, and thus adds its namespace to the additional namespaces list:
override __.AdditionalNamespaces = [typeof<DateTime>.Namespace]The assert template to use. You can find the full list of assert templates in generators/Templates/_Assert*.liquid
The RationalNumbers generator uses the "AssertEqualWithin" template to allow for checking doubles for equality with a specific tolerance:
override __.AssertTemplate canonicalDataCase =
match canonicalDataCase.Expected.Type with
| JTokenType.Float -> "AssertEqualWithin"
| _ -> base.AssertTemplate(canonicalDataCase)The test file's format: either a module or a class.
The default behavior is virtually always what you want. Only in extremely rare cases would you want to change this.
The Grep generator defines itself as a test class to allow using setup and teardown methods:
override __.TestFileFormat = TestFileFormat.ClassSpecifies the test method's name.
The default behavior is usually what you want. Only in very special circumstances would you want to change this.
The Zipper generator does some test method name cleanup:
override __.RenderTestMethodName canonicalDataCase =
let testMethodName = base.RenderTestMethodName canonicalDataCase
testMethodName.Replace("Set_", "Set ")Indicates if the test method name that is used should use the full path, which means that for nested test cases, it will prepend the test method name with the parent(s) description(s).
The default behavior is usually what you want. Only in cases where there would otherwise be multiple test methods with the same name would you want to change this.
The KindergartenGarden generator uses full method names to prevent test method name clashes:
override __.UseFullMethodName _ = trueIndicates if the test method should be marked with the Skip = true property.
The default behavior is virtually always what you want. Only in extremely rare cases would you want to change this.
The Markdown exercises is a refactoring exercise, and as such all its tests should already be in a valid state. Therefore, it doesn't add the Skip property for any of its test methods:
override __.SkipTestMethod (_, _) = falseIt is possible that an existing exercise does not match the canonical data. It is OK to update the exercise stub and/or the exercise example to follow the canonical data!
Also, if you find an issue with one of the existing generators or test suites simply open up the generator that you would like to update, make your changes, and then run the generators.
This repository is coded against .NET Core 2.x. To run the generators all you need to do is run the following command in the generators directory:
dotnet run
This command will take all of the exercise generators that are defined, and generate all of the test cases for each exercise. We use reflection to get all of the exercises, so if you are adding a new test, the test will be automatically included when running the generator.
If you only need to run a single generator, you can do so by running the following command:
dotnet run -e <exercise>
Once the generator has been run, you can view the output of your generation by navigating to the test file for that exercise. As an example, the test suite for the Bob exercise can be found at:
exercises/bob/BobTests.fs
If you are satisfied with the output of your generator, we would love for you to submit a pull request! Please include your generator, updated test suite, and any other corresponding files that you may have changed.