Test Cases for Practice Exercises

Exercism has two kinds of exercises: practice and concept. Concept exercises are tailored to fit a specific teaching goal and their tests can be fitted accordingly.

Practice exercises are either made for the track, or they come from Exercism's general pool. The so-called problem specifications repository is a central hub, which gets updated regularly. Each track is asked to implement the changes from the specifications. These may be changes in doc files or test cases. This document focuses on updating test cases for the problem specification practice exercises.

Test case information

As an example, look at the files for the difference of squares exercise:

• canonical-data.json (test cases)
• description.md (documentation)
• metadata.toml (general information)

Canonical Data

The canonical-data.json file is the one of interest for test cases. It looks like this:

{
  "exercise": "difference-of-squares",
  "cases": [
    {
      "description": "Square the sum of the numbers up to the given number",
      "cases": [
      ...
        {
          "uuid": "e46c542b-31fc-4506-bcae-6b62b3268537",
          "description": "square of sum 1",
          "property": "squareOfSum",
          "input": {
            "number": 1
          },
          "expected": 1
        },
        ...
      ]
    },
    {
      "description": "Sum the squares of the numbers up to the given number",
      "cases": [
        {
          "uuid": "01d84507-b03e-4238-9395-dd61d03074b5",
          "description": "sum of squares 5",
          "property": "sumOfSquares",
          "input": {
            "number": 5
          },
          "expected": 55
        },
        ...

Tests are organized as cases, each case might contain a subgroup of cases. The example has the "parent cases" with the properties squareOfSum and sumOfSquares. Each of these has "sub-cases" that test different values for the property. The last case can be read as "the sumOfSquares property should resolve to 55 if the input is 5". Each of these sub-case has a unique uuid, but the description is not necessarily unique, keep that in mind.

Track implementation

Each exercise on the track has a test.toml file to document the implemented tests:

[e46c542b-31fc-4506-bcae-6b62b3268537]
description = "square of sum 1"

[9b3f96cb-638d-41ee-99b7-b4f9c0622948]
description = "square of sum 5"

...

[01d84507-b03e-4238-9395-dd61d03074b5]
description = "sum of squares 1"

The toml file contains the uuids and the descriptions from the currently implemented tests.

Update process

With Exercism's configlet tool it is possible to sync the toml file with the canonical data file. A report is generated to show which tests have not been implemented yet.

Test structure

The difference of squares tests look like this:

...
TEST_CASE("up_to_10")
{
    REQUIRE(3025 == difference_of_squares::square_of_sum(10));
    REQUIRE(385 == difference_of_squares::sum_of_squares(10));
    REQUIRE(2640 == difference_of_squares::difference(10));
}

TEST_CASE("up_to_100")
{
    REQUIRE(25502500 == difference_of_squares::square_of_sum(100));
    REQUIRE(338350 == difference_of_squares::sum_of_squares(100));
    REQUIRE(25164150 == difference_of_squares::difference(100));
}

If a new test is added to the canonical data, each require has to be compared to the implemented test. Comparing the toml file information with the canonical data and the tests is a very manual process.

Structure recommendation

To facilitate the work of maintaining test cases, the structure is easier to generate and check with the following structure:

TEST_CASE("%%description%%", "[%%property%%][%%uid%%]") {
        TYPE expected{%%expected%%};
	REQUIRE(%%exercise_slug%%::%%property%%(%%inputs%%) == expected);
}

While the specific implementation can be very different from test to test, it generally should include the description, the property and the uuid in the TEST_CASE line, to facilitate (future) automation. Please do not use line breaks in the description, as this might mess with the test result display on the website.

Handling of old test scenarios

While updating old test files, it might be helpful to restructure them to make updates easier in the future.

Test creation helper tool

There is a branch, that has a creation helper tool in the bin directory, that might be helpful.