Skip to content

--c Flag for Continuing Training Not Implemented in MONAI nnUNet Runner #8237

@Kirscher

Description

@Kirscher

Describe the bug

I encountered an issue while using the MONAI nnUNetV2Runner. The --c flag, which is mentioned in the documentation and associated with continuing training from the latest checkpoint, does not seem to be implemented in the nnUNetV2Runner code. Including this flag in the command leads to an error indicating that it is unrecognized.

To Reproduce

Run the following command:
python -m monai.apps.nnunet nnUNetV2Runner train_single_model --input_config "input.yaml" --config "3d_fullres" --fold 1 --c
Observe the error:
error: unrecognized arguments: True

Expected behavior
The --c flag should:

Be recognized as a valid argument.
Enable continuation of training from the latest checkpoint without manually specifying a checkpoint file.

Environment

================================
Printing MONAI config...
================================
MONAI version: 1.4.0+18.g13b96aed
Numpy version: 1.26.4
Pytorch version: 2.5.1+cu124
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 13b96aedc48ad2da16149490b06a1a6bd8361335
MONAI __file__: /home/<username>/tutorials/venv/lib/python3.12/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.0
Nibabel version: 5.3.2
scikit-image version: 0.24.0
scipy version: 1.14.1
Pillow version: 11.0.0
Tensorboard version: 2.18.0
gdown version: 5.2.0
TorchVision version: 0.20.1+cu124
tqdm version: 4.67.0
lmdb version: 1.5.1
psutil version: 6.1.0
pandas version: 2.2.3
einops version: 0.8.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 2.18.0
pynrrd version: 1.1.1
clearml version: 1.16.5

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 24.04.1 LTS
Platform: Linux-6.8.0-49-generic-x86_64-with-glibc2.39
Processor: x86_64
Machine: x86_64
Python version: 3.12.3
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/home/kirscher/.config/Code/logs/20241125T093256/ptyhost.log', fd=43, position=434, mode='a', flags=33793), popenfile(path='/snap/code/176/usr/share/code/v8_context_snapshot.bin', fd=103, position=0, mode='r', flags=32768)]
Num physical CPUs: 12
Num logical CPUs: 24
Num usable CPUs: 24
CPU usage (%): [46.3, 42.1, 61.7, 35.9, 46.0, 64.4, 31.0, 68.1, 41.3, 56.4, 62.3, 26.9, 35.1, 17.1, 48.6, 49.4, 34.1, 34.5, 38.8, 50.7, 21.4, 84.8, 32.6, 17.7]
CPU freq. (MHz): 3514
Load avg. in last 1, 5, 15 mins (%): [22.6, 23.9, 24.1]
Disk usage (%): 94.2
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 62.6
Available memory (GB): 41.7
Used memory (GB): 18.1

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 12.4
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90']
GPU 0 Name: NVIDIA RTX A4500
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 56
GPU 0 Total memory (GB): 19.7
GPU 0 CUDA capability (maj.min): 8.6

Additional context
Using the standard nnUNetv2_train command outside of the MONAI runner framework bypasses the issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions