All predictions place in top left-hand corner [ RTX 3*** does NOT work with TensorFlow 1.x! == odd errors! Please use deeplabcutcore ]

[Ejdrup]

OS: Ubuntu 20.10 (Pop!_OS)
DeepLabCut Version: 2.1.10.2
CUDA: 10.0
cudnn: 7.6.5
GPU: RTX 3090
Tensorflow 1.13.1
Anaconda env used: DLC-GPU

After setting a new Ubuntu system and installing a fresh DLC-GPU, I've encountered an issue with GPU. Initially I received an error training, but managed to fix it following this issue, as I'm using RTX 3090: #1017

However, even though I'm able to progress through training without any issues, when I evaluate the network I get an error of roughly 500 pixels in both train and test despite running 100,000 iterations and plateuing at a loss of 0.03. However, all labels are placed in the top left-hand corner. I've tried starting a new project and relabeling, but to no avail. I've also attached a photo. Interestingly, after the first 1000 iterations the terminal outputs an astronomically high error, then drops to 0.12 at 2000.

I've previously used DLC without issues. I've even processed videos from an identical setup using older versions of DLC on a windows machine with great results.

Here's the terminal output:

Running  DLC_resnet_50_GUI_Salient_SNcGiMar12shuffle1_100000  with # of trainingiterations: 100000
Initializing ResNet

Analyzing data...

80it [00:01, 55.81it/s]

Done and results stored for snapshot:  snapshot-100000

/home/aske/anaconda3/envs/DLC-GPU/lib/python3.7/site-packages/deeplabcut/pose_estimation_tensorflow/evaluate.py:844: 
RuntimeWarning: Mean of empty slice

  RMSEpcutoff.iloc[testIndices].values.flatten()

/home/aske/anaconda3/envs/DLC-GPU/lib/python3.7/site-packages/deeplabcut/pose_estimation_tensorflow/evaluate.py:847:
 RuntimeWarning: Mean of empty slice

  RMSEpcutoff.iloc[trainIndices].values.flatten()

Results for 100000  training iterations: 95 1 train error: 499.12 pixels. Test error: 489.67  pixels.

With pcutoff of 0.6  train error: nan pixels. Test error: nan pixels

Thereby, the errors are given by the average distances between the labels by DLC and the scorer.

[MMathisLab]

can you confirm you have no labels in the top left corner your images when you run "check_labels" on your dataset ...?

also, GPU: RTX 3090 does NOT support Tensorflow 1! #944 #1119 will send you to use our TF2 code base, which is currently on the main branch of deeplabcutcore

[Ejdrup]

Doesn't #1017 solve that? I ran the following code:

pip install git+https://github.com/DeepLabCut/DeepLabCut-core.git@tf2.2alpha
pip install tf_slim

I can confirm I have no labels in the top left corner under .../labeled-data/_videofile__labeled. They are all correctly placed.

[MMathisLab]

^^ also note that now one can also just use pip install git+https://github.com/DeepLabCut/DeepLabCut-core.git or (in a few mins, pip install deeplabcutcore 👯‍♀️

@Ejdrup -- it must be you have TF1.x installed; we confirm this happens with TF 1, but NOT TF2!

[Ejdrup]

Cool. Would I need a fresh install of DLC-GPU environ, or can I just pip in the old environment?

[MMathisLab]

in the same env you can run pip uninstall tensorflow then pip install tensorflow==2.4 and pip install deeplabcutcore

(or also of course you can just make a new one)

[Ejdrup]

This is brand new setup, and we already have way too many cross-dependencies trying to load different cudnn and cuda versions, so we're just doing a fresh install. Which Ubuntu flavor do you guys use? It's primarily going to be a tracking workstations anyway, so trying to mimic your environment as much as possible.

[MMathisLab]

NOTE: we now support 3090 and TensorFlow 2! Please install a new conda file, you can direct download from deeplabcut.org - this comes with 2.2rc3 installed inside the conda, the latest tensorflow, and works with the latest CUDA.

see the blog: http://blog.deeplabcut.org/ for the new code base highlights