Testing: sct_testing

sct_testing

The SCT CLI script sct_testing used to be a custom testing framework that SCT used prior to the adoption of pytest. The custom framework has since been phased out, and SCT now uses an entirely pytest-based test suite. The actual sct_testing command has been preserved, though at this point it is simply a wrapper for pytest.

The old (outdated) Wiki page description for sct_testing is preserved below:

Introduction

sct_testing is a custom testing framework that SCT created before we started using pytest. This is how it's used:

# Run all `sct_testing` tests
sct_testing

# Run the `sct_testing` tests for a specific CLI script (e.g. `sct_deepseg_sc`)
sct_testing --function sct_deepseg_sc

How it works

For each CLI script offered by SCT, there is a corresponding test file. For example, sct_analyze_lesion corresponds to testing/test_sct_analyze_lesion.py, which looks like this:

def init(param_test):
    """
    Initialize class: param_test
    """
    # initialization
    default_args = ['-m t2/t2_seg-manual.nii.gz -s t2/t2_seg-manual.nii.gz']

    # assign default params
    if not param_test.args:
        param_test.args = default_args

    return param_test


def test_integrity(param_test):
    """
    Test integrity of function
    """
    # Simply check if output pkl file exists
    if os.path.exists('t2_seg-manual_analyzis.pkl'):
        param_test.output += '--> PASSED'
    else:
        param_test.status = 99
        param_test.output += '\nOutput file does not exist.'
    return param_test

When sct_testing is called, it does the following:

1. Finds the test file corresponding to whichever script was specified.
   • e.g. --function sct_analyze_lesion --> testing/test_sct_analyze_lesion.py.
2. It then calls the init function.
3. Then, it calls the CLI script using whichever arguments are in default_args.
   • e.g. sct_analyze_lesion default_args[0] becomes sct_analyze_lesion -m t2/t2_seg-manual.nii.gz -s t2/t2_seg-manual.nii.gz
4. Finally, it calls the test_integrity function to check the output of the CLI script.

Migration from sct_testing to pytest

We are in the process of removing sct_testing and replacing it with pytest. For now, we have very lightly wrapped sct_testing using a "backwards compatibility" test:

@pytest.mark.script_launch_mode('subprocess')
def test_sct_deepseg_sc_backwards_compat(script_runner):
    ret = script_runner.run('sct_testing', '--function', 'sct_deepseg_sc')
    logger.debug(f"{ret.stdout}")
    logger.debug(f"{ret.stderr}")
    assert ret.success
    assert ret.stderr == ''

This allows us to run the older tests using the newer pytest framework.

However, this is just temporary. Eventually the following should happen:

• Remove each of the original sct_testing tests.
• Remove each "backwards compatibility" wrapper test.
• Add new pytest tests that replicate the same functionality in the old test.
• Remove spinalcordtoolbox/scripts/sct_testing.py itself.