Merge branch 'master' into mhb/update-force-index-micr-ct

ivadomed · Nov 16, 2022 · d134642 · d134642
2 parents b3c7a8c + 5c59781
commit d134642
Show file tree

Hide file tree

Showing 9 changed files with 399 additions and 283 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,70 @@
+## v2.9.7 (2022-10-31)
+[View detailed changelog](https://github.com/ivadomed/ivadomed/compare/v2.9.6...release)
+
+**FEATURE**
+
+ - feat: update default args for `wandb.login()`.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1193)
+ - Add Config Parameter to Disable Validation When Loading BIDS Info.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1168)
+ - Add auto disk cache capability to mri2d and mri3d dataset classes.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1121)
+ - Segment 2D images without patches.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1101)
+
+**CI**
+
+ - chore: add an upper bound version specifier for pandas to prevent breaking the tests.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1194)
+ - Drop testing on macOS 10.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1190)
+ - Only run tests on code changes.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1186)
+ - chore: upgrade run_tests workflow.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1146)
+ - Don't install ivadomed just to run pre-commit checks..  [View pull request](https://github.com/ivadomed/ivadomed/pull/1145)
+
+**BUG**
+
+ - chore: add an upper bound version specifier for pandas to prevent breaking the tests.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1194)
+ - Drop testing on macOS 10.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1190)
+ - Resolve imageio v2->v3 imread deprecation warnings.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1181)
+ - fix: adapt the filenames of the predictions in pred_masks as per target_suffix.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1173)
+ - Fix long evaluation time on microscopy images.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1081)
+
+**INSTALLATION**
+
+ - chore: update installation for ivadomed tutorials.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1200)
+ - Remove deprecated torch and dev installation instructions.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1179)
+ - Get Rid of Python 3.6.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1149)
+ - Support python3.10.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1137)
+
+**DOCUMENTATION**
+
+ - fix: update link to contribution guidelines.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1196)
+ - Tests README: Add instructions to install testing-related packages.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1180)
+ - Remove deprecated torch and dev installation instructions.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1179)
+ - Add description and default values to some parameters..  [View pull request](https://github.com/ivadomed/ivadomed/pull/1174)
+ - Add Note in WandB.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1171)
+
+**DEPENDENCIES**
+
+ - chore: add an upper bound version specifier for pandas to prevent breaking the tests.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1194)
+ - Support python3.10.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1137)
+
+**ENHANCEMENT**
+
+ - feat: update default args for `wandb.login()`.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1193)
+ - Only run tests on code changes.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1186)
+ - Transformation on Subvolume for mri3d_subvolume_segmentation_dataset.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1169)
+ - Support python3.10.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1137)
+ - Add syntax highlighting and improve flow of the Colab tutorials.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1127)
+ - Add auto disk cache capability to mri2d and mri3d dataset classes.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1121)
+
+**TESTING**
+
+ - Drop testing on macOS 10.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1190)
+ - Only run tests on code changes.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1186)
+ - Add syntax highlighting and improve flow of the Colab tutorials.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1127)
+
+**REFACTORING**
+
+ - feat: update default args for `wandb.login()`.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1193)
+ - Convert pred data to uint8 prior to imwrite png.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1185)
+ - Minor correction to unsupported file extension error message.  [View pull request](https://github.com/ivadomed/ivadomed/pull/1177)
+
 ## v2.9.6 (2022-06-02)
 [View detailed changelog](https://github.com/ivadomed/ivadomed/compare/v2.9.5...release)
 

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -1,8 +1,22 @@
 Contributing to ivadomed
 ========================
 
+General Guidelines
+++++++++++++++++++
+
 Thank you for your interest in contributing to ivadomed! This project uses the following pages to guide new contributions:
 
-  * The `ivadomed GitHub repository <https://github.com/ivadomed/ivadomed>`_ is where the source code for the project is maintained, and where new contributions are submitted to.
-  * The `NeuroPoly Contributing Guidelines <https://intranet.neuro.polymtl.ca/software-development/contributing>`_ provide instructions for development workflows, such as reporting issues or submitting pull requests.
-  * The `ivadomed Developer Wiki <https://github.com/ivadomed/ivadomed/wiki>`_ acts as a knowledge base for documenting internal design decisions specific to the ivadomed codebase. It also contains step-by-step walkthroughs for common ivadomed maintainer tasks.
+  * The `ivadomed GitHub repository <https://github.com/ivadomed/ivadomed>`_
+    is where the source code for the project is maintained, and where new
+    contributions are submitted to. We welcome any type of contribution
+    and recommend setting up ``ivadomed`` by following the Contributor
+    or Developer installation as instructed below before proceeding
+    towards any contribution.
+
+  * The `NeuroPoly Contributing Guidelines <https://intranet.neuro.polymtl.ca/geek-tips/contributing.html>`_ 
+    provide instructions for development workflows, such as reporting issues or submitting pull requests.
+
+  * The `ivadomed Developer Wiki <https://github.com/ivadomed/ivadomed/wiki>`_
+    acts as a knowledge base for documenting internal design decisions specific
+    to the ivadomed codebase. It also contains step-by-step walkthroughs for
+    common ivadomed maintainer tasks.
diff --git a/docs/source/configuration_file.rst b/docs/source/configuration_file.rst
@@ -911,6 +911,11 @@ Split Dataset
         }
     }
 
+.. note::
+    .. line-block::
+            The fraction of the dataset used as validation set will correspond to ``1 - train_fraction - test_fraction``.
+            For example: ``1 - 0.6 - 0.2 = 0.2``.
+
 
 Training Parameters
 -------------------

diff --git a/ivadomed/config_manager.py b/ivadomed/config_manager.py
@@ -1,58 +1,61 @@
 import json
 import collections.abc
+from typing import Dict, List, Any, KeysView, Union
+
 from loguru import logger
 from pathlib import Path
 from ivadomed import utils as imed_utils
 from ivadomed.keywords import ConfigKW, LoaderParamsKW, SplitDatasetKW, DataTestingKW
 import copy
 
 
-def update(d, u):
+def update(source_dict: dict, destination_dict: dict) -> dict:
     """Update dictionary and nested dictionaries.
 
     Args:
-        d (dict): Source dictionary that is updated by destination dictionary.
-        u (dict): Destination dictionary.
+        source_dict (dict): Source dictionary that is updated by destination dictionary.
+        destination_dict (dict): Destination dictionary.
 
     Returns:
         dict: updated dictionary
     """
-    for k, v in u.items():
-        if isinstance(v, collections.abc.Mapping):
-            d[k] = update(d.get(k, {}), v)
+    for key, value in destination_dict.items():
+        if isinstance(value, collections.abc.Mapping):
+            source_dict[key] = update(source_dict.get(key, {}), value)
         else:
             # If source dictionary has keys that the destination dict doesn't have, keep these keys
-            if k in d and isinstance(d[k], collections.abc.Mapping) and not isinstance(v, collections.abc.Mapping):
+            if key in source_dict and isinstance(source_dict[key], collections.abc.Mapping) and not isinstance(value,
+                                                                                                               collections.abc.Mapping):
                 pass
             else:
-                d[k] = v
-    return d
+                source_dict[key] = value
+    return source_dict
 
 
-def deep_dict_compare(source_dict, dest_dict, keyname=None):
+def deep_dict_compare(source_dict: dict, destination_dict: dict, keyname: str = None):
     """Compare and display differences between dictionaries (and nested dictionaries).
 
     Args:
         source_dict (dict): Source dictionary.
-        dest_dict (dict): Destination dictionary.
+        destination_dict (dict): Destination dictionary.
         keyname (str): Key name to indicate the path to nested parameter.
 
     """
-    for key in dest_dict:
+    for key in destination_dict:
         if key not in source_dict:
             key_str = key if keyname is None else keyname + key
-            logger.info(f'    {key_str}: {dest_dict[key]}')
+            logger.info(f'    {key_str}: {destination_dict[key]}')
 
         else:
-            if isinstance(dest_dict[key], collections.abc.Mapping):
+            if isinstance(destination_dict[key], collections.abc.Mapping):
                 if isinstance(source_dict[key], collections.abc.Mapping):
-                    deep_dict_compare(source_dict[key], dest_dict[key], key + ": ")
+                    deep_dict_compare(source_dict[key], destination_dict[key], key + ": ")
                 # In case a new dictionary appears in updated file
                 else:
-                    deep_dict_compare(source_dict, dest_dict[key], key + ": ")
+                    deep_dict_compare(source_dict, destination_dict[key], key + ": ")
 
 
-def load_json(config_path):
+def load_json(config_path: str) -> dict:
     """Load json file content
 
     Args:
@@ -67,7 +70,7 @@ def load_json(config_path):
     return default_config
 
 
-# To ensure retrocompatibility for parameter changes in configuration file
+# To ensure retro-compatibility for parameter changes in configuration file
 KEY_CHANGE_DICT = {'UNet3D': ConfigKW.MODIFIED_3D_UNET, 'bids_path': LoaderParamsKW.PATH_DATA,
                    'log_directory': ConfigKW.PATH_OUTPUT}
 KEY_SPLIT_DATASET_CHANGE_LST = ['method', 'center_test']
@@ -84,68 +87,114 @@ class ConfigurationManager(object):
         context_original (dict): Provided configuration file.
         config_updated (dict): Updated configuration file.
     """
-    def __init__(self, path_context):
-        self.path_context = path_context
-        self.key_change_dict = KEY_CHANGE_DICT
-        self.key_split_dataset_change_lst = KEY_SPLIT_DATASET_CHANGE_LST
+
+    def __init__(self, path_context: str):
+        """
+        Initialize the ConfigurationManager by validating the given path and loading the file.
+        Also load the default configuration file.
+
+        Args:
+            path_context (str): Path to configuration file.
+        """
+        self.path_context: str = path_context
+        self.key_change_dict: Dict[str, str] = KEY_CHANGE_DICT
+        self.key_split_dataset_change_lst: List[str] = KEY_SPLIT_DATASET_CHANGE_LST
         self._validate_path()
-        default_config_path = str(Path(imed_utils.__ivadomed_dir__, "ivadomed", "config", "config_default.json"))
-        self.config_default = load_json(default_config_path)
-        self.context_original = load_json(path_context)
-        self.config_updated = {}
+        default_config_path: str = str(Path(imed_utils.__ivadomed_dir__, "ivadomed", "config", "config_default.json"))
+        self.config_default: dict = load_json(default_config_path)
+        self.context_original: dict = load_json(path_context)
+        self.config_updated: dict = {}
 
     @property
-    def config_updated(self):
+    def config_updated(self) -> dict:
+        """
+        This function simply returns the attribute `_config_updated`.
+
+        Returns:
+            dict: `_config_updated` attribute
+        """
         return self._config_updated
 
     @config_updated.setter
-    def config_updated(self, config_updated):
+    def config_updated(self, config_updated: dict):
+        """
+        If config_updated is empty we copy the loaded configuration into it and apply some changes (changing keys name,
+        changing values, deleting key-value pair) to ensure retro-compatibility.
+        Sets the new config_updated to the attribute `_config_updated`.
+
+        Args:
+            config_updated (dict): The new configuration to set.
+        """
         if config_updated == {}:
-            context = copy.deepcopy(self.context_original)
+            context: dict = copy.deepcopy(self.context_original)
             self.change_keys(context, list(context.keys()))
-            config_updated = update(self.config_default, context)
-            self.change_keys_values(config_updated[ConfigKW.SPLIT_DATASET], config_updated[ConfigKW.SPLIT_DATASET].keys())
+            config_updated: dict = update(self.config_default, context)
+            self.change_keys_values(config_updated[ConfigKW.SPLIT_DATASET],
+                                    config_updated[ConfigKW.SPLIT_DATASET].keys())
 
-        self._config_updated = config_updated
+        self._config_updated: dict = config_updated
         if config_updated['debugging']:
             self._display_differing_keys()
 
-    def get_config(self):
+    def get_config(self) -> dict:
         """Get updated configuration file with all parameters from the default config file.
+
         Returns:
             dict: Updated configuration dict.
         """
         return self.config_updated
 
-    def change_keys(self, context, keys):
-        for k in keys:
+    def change_keys(self, context: Union[dict, collections.abc.Mapping], keys: List[str]):
+        """
+        This function changes the name of the keys of the context dictionary, that are also in the `key_change_dict`
+        attribute, to the values that are associated with them in the `key_change_dict` attribute.
+
+        Args:
+            context (Union[dict, collections.abc.Mapping]): The dictionary to change.
+            keys (List[str]): The keys in context to consider.
+        """
+        for key_to_change in keys:
             # Verify if key is still in the dict
-            if k in context:
-                if k == "NumpyToTensor":
-                    del context[k]
+            if key_to_change in context:
+                # If the key_to_change is "NumpyToTensor", remove it from the context.
+                if key_to_change == "NumpyToTensor":
+                    del context[key_to_change]
                     continue
-                v = context[k]
+                value_to_change: Any = context[key_to_change]
                 # Verify if value is a dictionary
-                if isinstance(v, collections.abc.Mapping):
-                    self.change_keys(v, list(v.keys()))
+                if isinstance(value_to_change, collections.abc.Mapping):
+                    self.change_keys(value_to_change, list(value_to_change.keys()))
                 else:
                     # Change keys from the key_change_dict
                     for key in self.key_change_dict:
                         if key in context:
                             context[self.key_change_dict[key]] = context[key]
                             del context[key]
 
-    def change_keys_values(self, config_updated, keys):
-        for k in self.key_split_dataset_change_lst:
-            if k in keys:
-                value = config_updated[k]
-                if k == 'method' and value == "per_center":
+    def change_keys_values(self, config_updated: dict, keys: List[str]):
+        """
+        This function sets DATA_TESTING->DATA_TYPE to "institution_id" if method value is per_center,
+        DATA_TESTING->DATA_VALUE to the value of center_test.
+        It is basically verifying some conditions and set values to the `config_updated`.
+
+        Args:
+            config_updated (dict): Configuration dictionary to update.
+            keys (List[str]): The keys to consider.
+        """
+        for key_to_change in self.key_split_dataset_change_lst:
+            if key_to_change in keys:
+                value: Any = config_updated[key_to_change]
+                # If the method is per_center, the data_testing->data_type value becomes "institution_id".
+                if key_to_change == 'method' and value == "per_center":
                     config_updated[SplitDatasetKW.DATA_TESTING][DataTestingKW.DATA_TYPE] = "institution_id"
-                if k == 'center_test' and \
+                # If [the key is center_test], [data_testing->data_type == "institution_id"] and [the value is not None]
+                # data_testing->data_type value becomes value of config_updated
+                if key_to_change == 'center_test' and \
                         config_updated[SplitDatasetKW.DATA_TESTING][DataTestingKW.DATA_TYPE] == "institution_id" and \
-                value is not None:
+                        value is not None:
                     config_updated[SplitDatasetKW.DATA_TESTING][DataTestingKW.DATA_VALUE] = value
-                del config_updated[k]
+                # Remove the value of the current key
+                del config_updated[key_to_change]
 
     def _display_differing_keys(self):
         """Display differences between dictionaries.

diff --git a/ivadomed/evaluation.py b/ivadomed/evaluation.py
@@ -104,7 +104,8 @@ def evaluate(bids_df, path_output, target_suffix, eval_params):
                 imed_inference.pred_to_png(painted_list,
                                            target_list,
                                            str(path_preds.joinpath(subj_acq)),
-                                           suffix="_pred_painted.png")
+                                           suffix="_pred_painted.png",
+                                           max_value=3) # painted data contain 3 float values [0.0, 1.0, 2.0, 3.0] corresponding to background, TP, FP and FN objects)
 
         # SAVE RESULTS FOR THIS PRED
         results_pred['image_id'] = subj_acq

diff --git a/ivadomed/inference.py b/ivadomed/inference.py
@@ -212,7 +212,7 @@ def pred_to_nib(data_lst: List[np.ndarray], z_lst: List[int], fname_ref: str, fn
     return nib_pred
 
 
-def pred_to_png(pred_list: list, target_list: list, subj_path: str, suffix: str = ''):
+def pred_to_png(pred_list: list, target_list: list, subj_path: str, suffix: str = '', max_value: int = 1):
     """Save the network predictions as PNG files with suffix "_target_pred".
 
     Args:
@@ -221,11 +221,12 @@ def pred_to_png(pred_list: list, target_list: list, subj_path: str, suffix: str
         subj_path (str): Path of the subject filename in output folder without extension
             (e.g. "path_output/pred_masks/sub-01_sample-01_SEM").
         suffix (str): additional suffix to append to the filename (e.g. "_pred.png")
+        max_value (int): Maximum mask value of the float mask to use during the conversion to uint8.
     """
     for pred, target in zip(pred_list, target_list):
         filename = subj_path + target + suffix
         data = pred.get_fdata()
-        imageio.imwrite(filename, data, format='png')
+        imageio.imwrite(filename, (data*255/max_value).astype(np.uint8), format='png')
 
 
 def process_transformations(context: dict, fname_roi: str, fname_prior: str, metadata: dict, slice_axis: int,

diff --git a/ivadomed/version.txt b/ivadomed/version.txt
@@ -1 +1 @@
-2.9.6
+2.9.7