Isonet support NVIDIA A100

[cron-weasley]

Dear Author，
First thanks a lot for this powerful software !
I have a question: Is IsoNet support CUDA 11.2 and NVIDIA A100 with tensorflow-gpu_2.7?
I install isonet with conda python3.9 and tensorflow-gpu_2.7+cuda11.2+NVIDIA A100 get this error:

(py39) [root@Isonet]$ isonet.py refine subtomo.star --gpuID 0,1,2,3 --iterations 30 --noise_start_iter 10,15,20,25 --noise_level 0.05,0.1,0.15,0.2
04-16 22:14:09, INFO
######Isonet starts refining######

04-16 22:14:27, INFO Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
04-16 22:14:27, INFO Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
04-16 22:14:27, INFO NumExpr defaulting to 8 threads.
04-16 22:14:30, WARNING The results folder already exists before the 1st iteration
The old results folder will be renamed (to results~)
04-16 22:14:50, INFO Done preperation for the first iteration!
04-16 22:14:50, INFO Start Iteration1!
/data1/apps/miniconda3/envs/py39/lib/python3.9/site-packages/keras/optimizer_v2/adam.py:105: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(Adam, self).init(name, **kwargs)
/data1/apps/miniconda3/envs/py39/lib/python3.9/site-packages/keras/engine/functional.py:1410: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
layer_config = serialize_layer_fn(layer)
04-16 22:14:54, INFO Noise Level:0.0
2022-04-16 22:15:06.826516: F tensorflow/stream_executor/cuda/cuda_driver.cc:153] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error

Thanks a lot!

[procyontao]

Hi,

We only tested tensorflow2.5. and python3.6 on A100 GPUs. IsoNet works fine with A100.

Could you run the command with parameter: "--log_level debug" and see what is the error message?

[cron-weasley]

Hi,

We only tested tensorflow2.5. and python3.6 on A100 GPUs. IsoNet works fine with A100.

Could you run the command with parameter: "--log_level debug" and see what is the error message?

Dear procyontao,
Thanks a lot!
And could you tell me which cuda version and which NVIDIA A100 driver did you use?

[cron-weasley]

I sloved the problem.
Thanks procyontao!
I use python 3.8 with miniconda
and install:
pip install imageio==2.10.5 numpy==1.19.2
then install
pip install tensorflow-gpu==2.5.0
pip install -r requirements.txt
And Isonet run successfully.

[proteincommandr]

Hi all,
I also required TensorFlow v2.6.0 to run IsoNet properly on A100 GPUs.
I figured it would be nice to provide some GPU benchmarks for the tutorial dataset

I used TF2.6.0 Cuda11.6 IsoNet0.1 on the three tutorial tomograms

System Time/step speedup
2x 2070S 950ms 1
4x2080TI 700ms ~1.4
4x1080TI 900ms ~1
1xRTX8000 1000ms ~1
2xRTX8000 700ms ~1.3
4xA100 270ms ~3.5

Cheers

[procyontao]

Hi proteincommandr,

Thank you for providing the GPU benchmarks. We do not even afford that many types of GPU for a speed test.

One question, did you consider the differences in batch_size (i.e. number of subtomograms processed in one step)?
the default relation between number of GPUs and default batch_size is listed below:
nGPUs batch_size
1 4
2 4
3 6
4 8
5 10
6 12
7 14
8 16

If you do not specify the batch size in your command, your list should be:
System Time/step speedup batch_size speedup_persubtomo
2x 2070S 950ms 1 4 0.5
4x2080TI 700ms ~1.4 8 1.4
4x1080TI 900ms ~1 8 1
1xRTX8000 1000ms ~1 4 0.5
2xRTX8000 700ms ~1.3 4 0.65
4xA100 270ms ~3.5 8 3.5