automated testing

[drcandacemakedamoore]

I suggest you will be better of with testing in your CI. I wrote an example for your package (https://github.com/drcandacemakedamoore/reviewmirp/blob/master/.github/workflows/on-commit.yml) if you like it we can make a PR from this fork; if you envision a different approach go for it; but I see no more normative and efficient way to handle testing that putting it in the CI.

[alexzwanenburg]

Thanks for preparing a continuous integration script. My main concern is that running the full test suite currently requires at least 15 minutes on a high-end system (excluding setting up the environment). GitHub will likely take longer.

Currently, the test suite is run as part of version releases, see #64 for example. However, I can see the benefit of additionally running CI on (a subset of) the full test suite. This will also help identify issues when dependencies are making changes to their API.

I have opened a new branch for the PR.

[alexzwanenburg]

List of test durations:

183.87s call     test/documentation_example_test.py::test_extract_features_examples
133.61s call     test/perturbation_test.py::test_translation_perturbation_multiple
110.72s call     test/dicom_mr_test.py::test_bias_field_correction_t1
88.12s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_gabor_filter
58.53s call     test/readme_example_test.py::test_example_feature_extraction
35.92s call     test/perturbation_test.py::test_rotation_perturbation_multiple
34.10s call     test/ibsi_1_test.py::test_ibsi_1_chest_config_a
31.68s call     test/perturbation_test.py::test_perturbation_roi_randomisation
31.49s call     test/parallel_feature_extraction_test.py::test_parallel_feature_extraction
29.80s call     test/perturbation_test.py::test_pertubation_distance_change_multiple
28.02s call     test/perturbation_test.py::test_pertubation_fraction_change_multiple
23.70s call     test/perturbation_test.py::test_perturbation_roi_randomisation_rotation
23.58s call     test/ibsi_1_test.py::test_ibsi_1_chest_config_b
22.24s call     test/perturbation_test.py::test_noise_perturbation
18.44s call     test/basic_test.py::test_xml_configurations
18.23s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_simoncelli_filter
17.00s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_daubechies_filter
16.63s call     test/basic_test.py::test_edge_cases_basic_pipeline
16.15s call     test/ibsi_1_test.py::test_ibsi_1_chest_config_c
16.12s call     test/dicom_mr_test.py::test_bias_field_correction_t2
15.91s call     test/ibsi_1_test.py::test_ibsi_1_chest_config_e
14.17s call     test/ibsi_1_test.py::test_ibsi_1_chest_config_d
13.49s call     test/perturbation_test.py::test_perturbation_distance_grow
12.54s call     test/perturbation_test.py::test_perturbation_distance_shrink
12.50s call     test/perturbation_test.py::test_perturbation_fraction_shrink
12.49s call     test/perturbation_test.py::test_translation_perturbation
12.26s call     test/perturbation_test.py::test_rotation_perturbation
11.89s call     test/perturbation_test.py::test_perturbation_fraction_growth
11.56s call     test/dl_preprocessing_test.py::test_extract_image_crop
9.49s call     test/read_image_and_mask_test.py::test_read_dicom_image_and_mask_stack
8.07s call     test/intensity_scaling_test.py::test_intensity_scaling
7.91s call     test/read_image_and_mask_test.py::test_read_dicom_image_and_mask_modality_specific
7.19s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_laws_filter
7.18s call     test/parallel_feature_extraction_test.py::test_parallel_dl_preprocessing
6.70s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_laplacian_of_gaussian_filter
6.67s call     test/dicom_rtdose_test.py::test_basic_rtdose_feature_extraction
6.06s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_mean_filter
5.95s call     test/ibsi_2_feature_test.py::test_ibsi_2_config_none
5.66s call     test/dicom_pet_test.py::test_basic_pet_feature_extraction
5.63s call     test/dl_preprocessing_test.py::test_normalisation_quantile_range
5.55s call     test/dl_preprocessing_test.py::test_normalisation_range
5.51s call     test/dl_preprocessing_test.py::test_normalisation_relative_range
5.24s call     test/extract_image_parameters_test.py::test_extract_image_parameters_dicom
4.61s call     test/import_image_and_mask_test.py::test_multiple_image_and_mask_import_flat
4.04s call     test/dl_preprocessing_test.py::test_normalisation_standardisation
3.67s call     test/import_image_and_mask_test.py::test_multiple_image_and_mask_import
3.06s call     test/basic_test.py::test_orientation
3.02s call     test/ibsi_2_response_map_test.py::test_ibsi_2_gabor_filter
2.96s call     test/import_image_test.py::test_multiple_image_import_flat
2.64s call     test/dicom_mr_test.py::test_basic_mr_t1_feature_extraction
2.18s call     test/dicom_mr_test.py::test_basic_mr_t2_feature_extraction
2.06s call     test/import_mask_test.py::test_multiple_mask_import_flat
2.06s call     test/documentation_example_test.py::test_deeplearning_preprocessing
2.05s call     test/import_image_and_mask_test.py::test_single_image_and_mask_import_flat
1.97s call     test/import_image_and_mask_test.py::test_failure_multiple_image_and_mask_import_data_xml
1.96s call     test/import_image_and_mask_test.py::test_single_image_and_mask_import
1.94s call     test/import_image_and_mask_test.py::test_failure_multiple_image_and_mask_import
1.92s call     test/read_image_and_mask_test.py::test_read_dicom_image_and_mask_data_xml
1.91s call     test/readme_example_test.py::test_example_deep_learning_preprocessing
1.85s call     test/ibsi_1_test.py::test_ibsi_1_digital_phantom
1.76s call     test/import_image_test.py::test_multiple_image_import
1.70s call     test/import_image_test.py::test_single_image_import_flat
1.55s call     test/non_ibsi_compliant_filter_test.py::test_logarithm_transformation_filter
1.54s call     test/non_ibsi_compliant_filter_test.py::test_square_transformation_filter
1.43s call     test/dicom_ct_test.py::test_basic_ct_feature_extraction
1.42s call     test/non_ibsi_compliant_filter_test.py::test_exponential_transformation_filter
1.41s call     test/non_ibsi_compliant_filter_test.py::test_square_root_transformation_filter
1.29s call     test/import_image_test.py::test_single_image_import
1.07s call     test/read_image_and_mask_test.py::test_read_numpy_image_and_mask_stack
0.88s call     test/readme_example_test.py::test_example_image_metadata
0.84s call     test/import_mask_test.py::test_multiple_mask_import
0.70s call     test/ibsi_2_response_map_test.py::test_ibsi_2_laws_filter
0.51s call     test/read_image_and_mask_test.py::test_read_itk_image_and_mask
0.34s call     test/import_mask_test.py::test_single_mask_import
0.34s call     test/read_image_and_mask_test.py::test_read_numpy_image_and_mask
0.30s call     test/extract_mask_labels_test.py::test_extract_mask_labels_rtstruct
0.26s call     test/import_mask_test.py::test_single_mask_import_flat
0.24s call     test/ibsi_2_response_map_test.py::test_ibsi_2_haar_filter
0.23s call     test/ibsi_2_response_map_test.py::test_ibsi_2_coifflet_filter
0.22s call     test/ibsi_2_response_map_test.py::test_ibsi_2_daubechies_filter
0.22s call     test/read_image_and_mask_test.py::test_read_generic_image_and_mask_modality_specific
0.21s call     test/documentation_example_test.py::test_image_metadata_extraction
0.19s call     test/ibsi_2_response_map_test.py::test_ibsi_2_simoncelli_filter
0.15s call     test/ibsi_2_response_map_test.py::test_ibsi_2_mean_filter
0.15s call     test/ibsi_2_response_map_test.py::test_ibsi_2_log_filter
0.13s call     test/extract_mask_labels_test.py::test_extract_mask_labels_generic
0.12s call     test/read_image_and_mask_test.py::test_read_numpy_image_and_mask_online
0.10s call     test/readme_example_test.py::test_example_retrieve_mask_labels
0.08s call     test/configuration_test.py::test_image_transformation_settings_configuration
0.06s call     test/documentation_example_test.py::test_mask_label_extraction
0.05s call     test/import_image_test.py::test_image_import_flat_poor_naming
0.03s call     test/export_image_and_mask_test.py::test_basic_ct_feature_extraction
0.03s call     test/extract_image_parameters_test.py::test_extract_image_parameters_default
0.01s call     test/mask_processing_test.py::test_mask_processing
0.01s call     test/import_mask_test.py::test_mask_import_flat_poor_naming
0.01s call     test/configuration_test.py::test_general_settings_configuration

[drcandacemakedamoore]

I think it's fine to not test everything, but there should be some critical group of tests that get run on auto- otherwise it's too easy, especially if someone is running away with a fork of their own, to just start slowly breaking everything without realizing it.

[alexzwanenburg]

I took the liberty of writing a CI workflow based on a standard GitHub workflow: .github/workflows/auto-test-package.yml.

This also checks against different OS and python versions (currently limited to 3.10 and 3.11 due to limitations in the ray package).

A Pull Request is no longer required.

[drcandacemakedamoore]

Great but if I recall Python 3.11 should throw you some problems, and Python 3.12 definitely did when I tested it. Look very carefully at each package, and the requirements before adding Python 3.11.

[alexzwanenburg]

Python 3.12 likely throws errors because its not supported by ray yet. Other versions seem to build correctly and test correctly: https://github.com/oncoray/mirp/actions/runs/8433681661.

The auto-ci test suite also includes a test that uses parallel processing.

[alexzwanenburg]

To do:

• Mark automated tests using custom pytest.mark decorators: https://docs.pytest.org/en/latest/example/markers.html#mark-examples