input image contains negative values and gpu memory issue

[f4bry]

Question/Support Request

Dear FastSurfer experts, I'd like to run the tool using singularity and a GPU. My script looks like:

singularity exec --nv \
                 --no-home \
                 -B /imaging/rawT1/sub-01/T1:/data \
                 -B //imaging/fastsurfer_output:/output \
                 -B /usr/local/freesurfer:/fs \
                  /mnt/wsdata/softwares/fastsurfer-gpu.sif \
                  /fastsurfer/run_fastsurfer.sh \
                 --fs_license /fs/license.txt \
                 --t1 /data/T1.nii.gz \
                 --sid sub-01 --sd /output \
                 --seg_only

I am getting the following error:

/fastsurfer/FastSurferCNN /
python3.8 eval.py --in_name /data/T1.nii.gz --out_name /output/sub-01/mri/aparc.DKTatlas+aseg.deep.mgz --conformed_name /output/sub-01/mri/orig.mgz --order 1 --network_sagittal_path /fastsurfer/checkpoints/Sagittal_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl --network_axial_path /fastsurfer/checkpoints/Axial_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl --network_coronal_path /fastsurfer/checkpoints/Coronal_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl --batch_size 8 --simple_run --run_viewagg_on check
Reading volume /data/T1.nii.gz
The input image is not conformed.
A conformed image must satisfy the following criteria:
 - Dimensions 256x256x256:        False
 - Voxel size 1x1x1:              False
 - Orientation LIA:               False
 - Dtype uint8:                   False
Conforming image to UCHAR, RAS orientation, and 1mm isotropic voxels

ERROR: Min value in input is below 0.0!

However, the T1.nii.gz volume looks good when I open it with other tools.
Do you know where that error could come from?
Thank you,
f4b

Environment

• FastSurfer Version: latest
• FreeSurfer Version: 6.0
• OS: Ubuntu 22.04.4 LTS
• CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
• GPU: GeForce RTX 3080 Lite Hash Rate

[m-reuter]

For some reason your image has negative values, which usually should not happen. Maybe it was pre-processed, for example, cubic interpolation can produce negative values. If they are just slightly below zero, you can clamp them to zero.

[f4bry]

Thanks!

It runs now, but I get the following error:

RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 9.78 GiB total capacity; 6.69 GiB already allocated; 265.00 MiB free; 7.00 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Is it possible to set max_split_size_mb running singularity?
Better opening a new issue?

Thank you,
f4b

[m-reuter]

Hi, this is a CUDA error and I don't think you can set these. It is strange that you have memory issues on a 10GB card. The question is what is already taking up the 7GiB. Are you running anything else on the GPU? When is this error happening (during the first segmentation network, or during view aggregation )? Oh, and what resolution is you input image?

[f4bry]

This is the error:

I am trying to set that variable, like

singularity exec --nv \
                 --no-home \
                 --env PYTORCH_CUDA_ALLOC_CONF='max_split_size_mb=1024'
......

Here is the error

Reading volume /data/pos_T1.nii.gz
The input image is not conformed.
A conformed image must satisfy the following criteria:
 - Dimensions 256x256x256:        False
 - Voxel size 1x1x1:              False
 - Orientation LIA:               False
 - Dtype uint8:                   False
Conforming image to UCHAR, RAS orientation, and 1mm isotropic voxels

Input:    min: 0.0  max: 4337.0
rescale:  min: 0.0  max: 3148.662  scale: 0.08098678105176103
Output:   min: 0.0  max: 255.0
Saving conformed original image to /output/sub-01/mri/orig.mgz
Cuda available: True, # Available GPUS: 1, Cuda user disabled (--no_cuda flag): False, --> Using device: cuda
Loading Axial
Successfully loaded Image from /data/pos_T1.nii.gz
Loading Axial Net from /fastsurfer/checkpoints/Axial_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl
Axial model loaded.
Traceback (most recent call last):
  File "eval.py", line 485, in <module>
    fastsurfercnn(options.iname, options.oname, options.conformed_name, use_cuda, small_gpu, logger, options)
  File "eval.py", line 315, in fastsurfercnn
    pred_prob = run_network(img_filename,
  File "eval.py", line 213, in run_network
    temp = model(images_batch)
  File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/fastsurfer/FastSurferCNN/models/networks.py", line 79, in forward
    decoder_output1 = self.decode1.forward(decoder_output2, skip_encoder_1, indices_1)
  File "/fastsurfer/FastSurferCNN/models/sub_module.py", line 305, in forward
    out_block = super(CompetitiveDecoderBlock, self).forward(concat_max)
  File "/fastsurfer/FastSurferCNN/models/sub_module.py", line 102, in forward
    x1 = self.prelu(x1_max)
  File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/venv/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1076, in forward
    return F.prelu(input, self.weight)
  File "/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 1500, in prelu
    return torch.prelu(input, weight)
RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 9.78 GiB total capacity; 6.94 GiB already allocated; 25.19 MiB free; 7.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Thanks!
f4b

[m-reuter]

The error happens right at the start.

• Are you using the latest release official docker image (version 2.2.0) converted to singularity?
• And also what is the resolution of your input image?
• Are you running anything else on the GPU? Maybe on the host system (window manager. ...)?

[f4bry]

The error happens right at the start.

• Are you using the latest release official docker image (version 2.2.0) converted to singularity?

That was the problem. I re-run singularity build fastsurfer-gpu.sif docker://deepmi/fastsurfer:latest and now it works! 3' 75'' sec to get till the end with --seg_only. Thank you very much!

• And also what is the resolution of your input image?
• Are you running anything else on the GPU? Maybe on the host system (window manager. ...)?