Very slow training with RTX 3090

[Marti-R]

OS: Win 10
CUDA: 11.1
Cudnn: 8.0.5.39 (Installed as described in https://www.reddit.com/r/tensorflow/comments/jsalkw/rtx_3090_and_tensorflow_for_windows_10_step_by/)
Driver: 461.40

Tensorflow: 2.4.0 and deeplabcutcore from https://github.com/DeepLabCut/DeepLabCut-core.git@tf2.2alpha in the quick-install anaconda environment DLC-GPU

Hello everyone,

I tried to train my models with a RTX 3090, but I quickly realized that the regular quick-install deeplabcut wont work anymore. In issue #944 there was a general outline on what to do to start training with deeplabcutcore and I followed these and got it somehow working. My issue now is though, that the above setup allows me to train but is extremely slow when doing so.
I have another setup with Ubuntu 18.04 and a RTX 2080Ti and this is about 5x faster. Is there some issue with the setup I described above that slows the training process down?

During the installation I noticed a few issues:
Just installing deeplabcutcore with pip will (in my case) led to the error that tensorflow.contrib could not be found. This is why I used the 2.2alpha branch with tf_slim, as described in #1024.
Installing the tf-nightly-gpu package will lead to a freeze as soon as I import deeplabcutcore (same installation as above). This is why I used tensorflow 2.4.0.

Can someone who is also using a RTX 3000 card tell me how to properly set this up? I was hoping to speed up the training compared to the other setup.

Thanks in advance for any suggestions you can give!

[MMathisLab]

hi @Marti-R plese open deeplabcutcore related issues here: https://github.com/DeepLabCut/DeepLabCut-core

also note TF2 is is in the main branch now (not just alpha; you might consider trying main; and also be sure you are using batch-sizing, etc correctly)