You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Dimension mismatch during validation between ground truth data and predictions
#2946
First off: great job with this project! Really do appreciate all of the effort that is put into this.
Now to the issue at hand. I am currently trying to train a model for segmenting lesions in DBT-volumes (https://www.cancerimagingarchive.net/collection/breast-cancer-screening-dbt/). To this purpose I have manually created ground truth segmentations of lesions and stored them as nifti-files. I followed the instructions on how to create a custom dataset which worked perfectly fine.
For the experiment, I've run the following:
>nnUNetv2_plan_and_preprocess -d 420 -c 3d_lowres -pl nnUNetPlannerResEncM -np 2 --verify_dataset_integrity --clean
>for f in {0..4}; do
nnUNetv2_train 420 3d_lowres $f -device cuda -p nnUNetResEncUNetMPlans -tr nnUNetTrainer_250epochs
done
It trains without any hiccups to 250 epochs, runs the validation and for folds 0, 2, 3 it fails during the validation step:
/.../
2025-11-22 11:00:17.367552: predicting DBT_181
2025-11-22 11:00:17.442365: DBT_181, shape torch.Size([1, 25, 738, 340]), rank 0
2025-11-22 11:00:26.782586: predicting DBT_192
2025-11-22 11:00:26.857468: DBT_192, shape torch.Size([1, 23, 776, 330]), rank 0
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/evaluation/evaluate_predictions.py", line 105, in compute_metrics
tp, fp, fn, tn = compute_tp_fp_fn_tn(mask_ref, mask_pred, ignore_mask)
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/evaluation/evaluate_predictions.py", line 81, in compute_tp_fp_fn_tn
tp = np.sum((mask_ref & mask_pred) & use_mask) ValueError: operands could not be broadcast together with shapes (1,60,2457,1996) (1,69,2457,1996)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/maia-user/jacob/envs/nnunetv2/bin/nnUNetv2_train", line 7, in
sys.exit(run_training_entry())
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/run/run_training.py", line 266, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/run/run_training.py", line 211, in run_training
nnunet_trainer.perform_actual_validation(export_validation_probabilities)
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1345, in perform_actual_validation
metrics = compute_metrics_on_folder(join(self.preprocessed_dataset_folder_base, 'gt_segmentations'),
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/evaluation/evaluate_predictions.py", line 143, in compute_metrics_on_folder
results = pool.starmap(
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value ValueError: operands could not be broadcast together with shapes (1,60,2457,1996) (1,69,2457,1996)
In summation: some predictions do not match the corresponding GT_segmentations' dimensions. Which is odd.
I've scoured the discussions/issues and only found one similar case (#2393, however, it seems they only ran into the issue during inference) but there was never a resolution to that thread.
Any help would be appreciated! I'm a bit of a newbie when it comes to this tooling, so am at loss for how to debug this.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Hi,
First off: great job with this project! Really do appreciate all of the effort that is put into this.
Now to the issue at hand. I am currently trying to train a model for segmenting lesions in DBT-volumes (https://www.cancerimagingarchive.net/collection/breast-cancer-screening-dbt/). To this purpose I have manually created ground truth segmentations of lesions and stored them as nifti-files. I followed the instructions on how to create a custom dataset which worked perfectly fine.
For the experiment, I've run the following:
It trains without any hiccups to 250 epochs, runs the validation and for folds 0, 2, 3 it fails during the validation step:
/.../
2025-11-22 11:00:17.367552: predicting DBT_181
2025-11-22 11:00:17.442365: DBT_181, shape torch.Size([1, 25, 738, 340]), rank 0
2025-11-22 11:00:26.782586: predicting DBT_192
2025-11-22 11:00:26.857468: DBT_192, shape torch.Size([1, 23, 776, 330]), rank 0
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/evaluation/evaluate_predictions.py", line 105, in compute_metrics
tp, fp, fn, tn = compute_tp_fp_fn_tn(mask_ref, mask_pred, ignore_mask)
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/evaluation/evaluate_predictions.py", line 81, in compute_tp_fp_fn_tn
tp = np.sum((mask_ref & mask_pred) & use_mask)
ValueError: operands could not be broadcast together with shapes (1,60,2457,1996) (1,69,2457,1996)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/maia-user/jacob/envs/nnunetv2/bin/nnUNetv2_train", line 7, in
sys.exit(run_training_entry())
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/run/run_training.py", line 266, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/run/run_training.py", line 211, in run_training
nnunet_trainer.perform_actual_validation(export_validation_probabilities)
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1345, in perform_actual_validation
metrics = compute_metrics_on_folder(join(self.preprocessed_dataset_folder_base, 'gt_segmentations'),
File "/home/maia-user/jacob/external/nnUNet/nnunetv2/evaluation/evaluate_predictions.py", line 143, in compute_metrics_on_folder
results = pool.starmap(
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/home/maia-user/jacob/envs/nnunetv2/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
ValueError: operands could not be broadcast together with shapes (1,60,2457,1996) (1,69,2457,1996)
In summation: some predictions do not match the corresponding GT_segmentations' dimensions. Which is odd.
I've scoured the discussions/issues and only found one similar case (#2393, however, it seems they only ran into the issue during inference) but there was never a resolution to that thread.
Any help would be appreciated! I'm a bit of a newbie when it comes to this tooling, so am at loss for how to debug this.
All reactions