Process killed

[TheJaeger]

Hello! First of all, thanks for pushing this out. I've been testing this on my end directly with the run_fastsurfer.py file. Unfortunately the process gets killed and I'm not sure why. I do suspect it may have to do with memory.

(fastsurfer) sid@sid-research:~/Repos/FastSurfer$ python run_fastsurfer.py --sid IAM_1116 --sd ~/Desktop/FastSurfer/Out/ --seg /home/sid/Desktop/FastSurfer/Out/IAM_1116/aparc+aseg_deep.mgz --t1 '/home/sid/Desktop/FastSurfer/In/IAM_1116/anat/IAM_1116_5_t1_mprage_sag_p2_iso__.nii.gz' --n_threads 16
Reading volume /home/sid/Desktop/FastSurfer/In/IAM_1116/anat/IAM_1116_5_t1_mprage_sag_p2_iso__.nii.gz
Conforming image to UCHAR, RAS orientation, and 1mm isotropic voxels
Input:    min: 0  max: 684
rescale:  min: 0.0  max: 454.86  scale: 0.5606120564569318
Output:   min: 0.0  max: 255.0
Loading Axial
Successfully loaded Image from /home/sid/Desktop/FastSurfer/In/IAM_1116/anat/IAM_1116_5_t1_mprage_sag_p2_iso__.nii.gz
Loading Sagittal
Successfully loaded Image from /home/sid/Desktop/FastSurfer/In/IAM_1116/anat/IAM_1116_5_t1_mprage_sag_p2_iso__.nii.gz
Loading Coronal.
Successfully loaded Image from /home/sid/Desktop/FastSurfer/In/IAM_1116/anat/IAM_1116_5_t1_mprage_sag_p2_iso__.nii.gz
Cuda available: True, # Available GPUS: 1, Cuda user disabled (--no_cuda flag): False, --> Using device: cuda
Loading Axial Net from ./checkpoints/Axial_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl
Axial model loaded.
--->Batch 0 Axial Testing Done.
--->Batch 1 Axial Testing Done.
--->Batch 2 Axial Testing Done.
--->Batch 3 Axial Testing Done.
--->Batch 4 Axial Testing Done.
--->Batch 5 Axial Testing Done.
--->Batch 6 Axial Testing Done.
--->Batch 7 Axial Testing Done.
--->Batch 8 Axial Testing Done.
--->Batch 9 Axial Testing Done.
--->Batch 10 Axial Testing Done.
--->Batch 11 Axial Testing Done.
--->Batch 12 Axial Testing Done.
--->Batch 13 Axial Testing Done.
--->Batch 14 Axial Testing Done.
--->Batch 15 Axial Testing Done.
Axial View Tested in 18.0291 seconds
Loading Coronal Net from ./checkpoints/Coronal_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl
Killed

This is running in a Python 3.6 environment on a system with Ryzen 2700x, GTX1080 and 16 GB RAM.

EDIT: Upon further investigation, it is indeed an out of memory error:

[Wed Jun 24 14:40:37 2020] Out of memory: Killed process 20341 (python) total-vm:31198980kB, anon-rss:11861040kB, file-rss:99992kB, shmem-rss:12288kB, UID:1000 pgtables:26172kB oom_score_adj:0

That translates to ~32 gigabytes of memory that the script requested. I'll try this on a system with larger memory.

[LeHenschel]

Hey,

first, it would be better if you used the shell script (run_fastsurfer.sh). The memory problem is most likely caused by the batch size (it is 16 by default). You can change it with the flag --batch (or --batch_size for the python script) e.g. set it to 1 to see if the problem goes away.

Best,
Leonie

[m-reuter]

This is probably resolved. In case reducing the batch size did not solve it, we can reopen.

[TheJaeger]

This fixed it, thanks!