Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3080 & 3090 coumpute capability 86 degraded performance after some updates #44116

Open
ibmua opened this issue Oct 17, 2020 · 5 comments
Open

3080 & 3090 coumpute capability 86 degraded performance after some updates #44116

ibmua opened this issue Oct 17, 2020 · 5 comments

Comments

@ibmua
Copy link

@ibmua ibmua commented Oct 17, 2020

This issue is apparent from the difference in performance in NGC containers https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow . For 20.08 se-resnext101 example training performance

python nvidia-examples/resnet50v1.5/main.py --arch=se-resnext101-32x4d --batch_size=64  --warmup_steps 200 --data_dir=/hdd/datasets/imagenet/tf/train/ --gpu_memory_fraction 0.95  --precision fp32  --results_dir=/toy/tmp/results_dir/   --mode=training_benchmark   --use_tf_amp --use_xla

(have to adapt directories)
on 3080 is around 370-400 img/sec. While on 20.09 container it's more like 115 img/sec. This is also similar for resnet-50 and most likely all other CNN benchmarks. This is not an issue with my setup, it's the same for other folks - you can view discussion at https://www.pugetsystems.com/labs/hpc/RTX3090-TensorFlow-NAMD-and-HPCG-Performance-on-Linux-Preliminary-1902/

@sanjoy
Copy link
Member

@sanjoy sanjoy commented Oct 22, 2020

Can you please report this on the NVIDIA developer forum?

CC @nluehr

We can circle back here if/when this is triaged down to an issue with the TF nightly and/or TF release builds.

@ibmua
Copy link
Author

@ibmua ibmua commented Oct 24, 2020

3090's performance on 20.10 tf1 ngc container is even 15-20% better than on 20.08, so I guess we can just agree that we should never use 20.08 container because it sucks and let go of this issue. I'll later also try to run some tests on 20.10 tf2 and report back.

Edit: seems like might be very different for different cases, though. Got to test more. Will report later.

@ibmua
Copy link
Author

@ibmua ibmua commented Nov 10, 2020

@ibmua
Copy link
Author

@ibmua ibmua commented Nov 12, 2020

(And the performance was pretty inconsistent, you better take a look.)

@ibmua
Copy link
Author

@ibmua ibmua commented Nov 25, 2020

Retested on 20.11 container. 3080 performance still effed up https://fsymbols.com/3080-3090-benchmarks/ It still has Cudnn 8.04 and same CUDA version as 20.10 container, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants