Testing: Writing new tests

We use pytest to verify that spinalcordtoolbox is working correctly. SCT has two main types of tests: API Tests, which test individual functions in SCT's Python API, and CLI Tests, which test SCT's CLI scripts.

For instructions on how to run the test suite, refer to the Running the test suite Wiki page.

General recommendations

Below are some general recommendations that apply to all kinds of tests:

• For input data, you can use real anatomical data from sct_testing_data/, or create dummy data using a Pytest fixture. Further details can be found on the Test data Wiki page.
• Parametrization can be used to repeat tests across multiple different test cases.
• Use the built-in pytest fixture tmp_path as an output folder (-ofolder in most SCT CLI scripts), or as the root directory for output filepaths (-o in most SCT CLI scripts). This ensures that no new files are left behind by your tests. Ideally, the test data directory (sct_testing_data/) should be left in the same state it was in prior to the test.
• Use logging in tests (don't print). (This lets SCT developers combine different logging levels with pytest's filtering options to better analyze test results.)
• It is a good idea to test the pass case and the fail case as well.
• Try not to repeat yourself. If you're using the same setup/initialization steps across multiple tests, consider grouping them into a fixture, or refactoring those tests into a single test using parametrization.

API tests for spinalcordtoolbox

API tests exist in the testing/api directory, and are grouped by module into test files called test_${module}.py.

Conceptually, you can think of these like SCT's unit tests, since they test individual functions in the spinalcordtoolbox Python API.

For API tests, please follow these guidelines:

• Unit tests should aim to be simple and specific.
• Unit tests should aim to avoid coupling/interdependence.
• Unit tests should not cause unexpected side effects (modify an input file, create unwanted artefacts on the filesystem, etc).

CLI tests for spinalcordtoolbox/scripts

CLI tests exist in the testing/cli/ folder, and are meant to verify the functionality of each of SCT's CLI scripts found within spinalcordtoolbox/scripts.

Conceptually, you can think of these tests as functional/integration tests, since each CLI script combines various functions from SCT's Python API.

For CLI tests, please follow these guidelines:

• Call the CLI script using import sct_function, then sct_function.main(args) (instead of using subprocess/sct.run). This allows things like pytest.raises to be used if checking for an expected exception.
• Tests should make a best effort at testing the main functionalities provided by the sct_function. You can refer to SCT's tutorials to get a sense for how each script is used in practice.