Getting [Error 38] in nnUNet_plan_and_preprocess stage

[Alaska47]

Hi,

I'm trying to train a brand new model with a custom dataset using nnUNet. I already converted my dataset into the appropriate format, and double-checked this using the --verify_dataset_integrity flag. However, during the preprocess stage, I'm running into an error.

transpose forward [0, 1, 2]
transpose backward [0, 1, 2]
Traceback (most recent call last):
  File "/anaconda/envs/azureml_py36/bin/nnUNet_plan_and_preprocess", line 33, in <module>
    sys.exit(load_entry_point('nnunet', 'console_scripts', 'nnUNet_plan_and_preprocess')())
  File "/nnUNet/nnunet/experiment_planning/nnUNet_plan_and_preprocess.py", line 128, in main
    exp_planner.run_preprocessing(threads)
  File "/nnUNet/nnunet/experiment_planning/experiment_planner_baseline_3DUNet.py", line 426, in run_preprocessing
    "gt_segmentations"))
  File "/anaconda/envs/azureml_py36/lib/python3.6/shutil.py", line 365, in copytree
    raise Error(errors)
shutil.Error: [('/nnUnet/nnUNet_raw_data_base/nnUNet_cropped_data/Task001_Brain/gt_segmentations/BRAIN_000.nii.gz', '/nnUnet/nnUNet_preprocessed/Task001_Brain/gt_segmentations/BRAIN_000.nii.gz', "[Errno 38] Function not implemented: '/nnUnet/nnUNet_raw_data_base/nnUNet_cropped_data/Task001_Brain/gt_segmentations/BRAIN_000.nii.gz'"), 
...

and this error repeats for every sample in the dataset.

I checked and see that /nnUnet/nnUNet_raw_data_base/nnUNet_cropped_data/Task001_Brain/gt_segmentations/BRAIN_000.nii.gz exists, /nnUnet/nnUNet_preprocessed/Task001_Brain/gt_segmentations/BRAIN_000.nii.gz exists so I'm not really sure what the problem could be here.

[FabianIsensee]

Hi,
this is very strange indeed. Could it be that some of your code files are corrupted? Can you please uninstall nnunet and re-install from the current master?
Best,
Fabian

[Alaska47]

Hi Fabian,

I was able to resolve the issue. Basically, I was running the code in a Microsoft Azure instance and there seems to be an issue with Azure machines when running shutil.copytree(). I was able to use the fix mentioned in https://stackoverflow.com/questions/51616058/shutil-copystat-fails-inside-docker-on-azure

I modified

nnUNet/nnunet/experiment_planning/experiment_planner_baseline_3DUNet.py

Lines 425 to 426 in b569f1d

	shutil.copytree(join(self.folder_with_cropped_data, "gt_segmentations"),
	join(self.preprocessed_output_folder, "gt_segmentations"))

to be

with mock.patch('os.listxattr', return_value=[]):
    shutil.copytree(join(self.folder_with_cropped_data, "gt_segmentations"), join(self.preprocessed_output_folder, "gt_segmentations"))

I also made the same change to

nnUNet/nnunet/experiment_planning/utils.py

Line 73 in b569f1d

shutil.copytree(join(input_folder, "labelsTr"), join(output_folder, "labelsTr"))

to be

with mock.patch('os.listxattr', return_value=[]):
    shutil.copytree(join(input_folder, "labelsTr"), join(output_folder, "labelsTr"))

Also just wanted to say, great project! I'm really impressed with the work and it proved to be very useful for me.

[FabianIsensee]

Hi, glad to hear you could solve it. I have never tested nnU-net on could instances and therefore not noticed this. I will link your solution in my documentation if that's okay with you. That way users of azure will know what to do if they run into the same issues
Best,
Fabian

[Alaska47]

Sounds good. Glad I could help!

[kirillmeisser]

Hi @Alaska47,

I also am trying to run the nnU-Net on an Azure instance. If you have time would you kindly explain how you managed to run the nnU-Net on Azure Machine Learning Studio? I am new to Azure and any help would be greatly appreciated!

Best,

Kirill

[FabianIsensee]

Hi Fabian,

I was able to resolve the issue. Basically, I was running the code in a Microsoft Azure instance and there seems to be an issue with Azure machines when running shutil.copytree(). I was able to use the fix mentioned in https://stackoverflow.com/questions/51616058/shutil-copystat-fails-inside-docker-on-azure

I modified

nnUNet/nnunet/experiment_planning/experiment_planner_baseline_3DUNet.py

Lines 425 to 426 in b569f1d

	shutil.copytree(join(self.folder_with_cropped_data, "gt_segmentations"),
	join(self.preprocessed_output_folder, "gt_segmentations"))

to be

with mock.patch('os.listxattr', return_value=[]):
    shutil.copytree(join(self.folder_with_cropped_data, "gt_segmentations"), join(self.preprocessed_output_folder, "gt_segmentations"))

I also made the same change to

nnUNet/nnunet/experiment_planning/utils.py

Line 73 in b569f1d

shutil.copytree(join(input_folder, "labelsTr"), join(output_folder, "labelsTr"))

to be

with mock.patch('os.listxattr', return_value=[]):
    shutil.copytree(join(input_folder, "labelsTr"), join(output_folder, "labelsTr"))

Also just wanted to say, great project! I'm really impressed with the work and it proved to be very useful for me.

@kirillmeisser this is what @Alaska47 used to fix the issue

Best,
Fabian