Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Getting a warning about Nvidia Tensor Cores #708

Open
yardencsGitHub opened this issue Sep 29, 2023 · 1 comment
Open

BUG: Getting a warning about Nvidia Tensor Cores #708

yardencsGitHub opened this issue Sep 29, 2023 · 1 comment
Labels
BUG Something isn't working

Comments

@yardencsGitHub
Copy link
Collaborator

Describe the bug
When running with a GPU that has Tensor Cores I am getting the following warning:

You are using a CUDA device ('NVIDIA GeForce RTX 4090') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision

To Reproduce
Simply run 'vak train' or 'vak predict' from the command line using the above GPU.

Desktop (please complete the following information):

  • Operating System: Windows
  • Version 1.0.0a2
@yardencsGitHub yardencsGitHub added the BUG Something isn't working label Sep 29, 2023
@NickleDave
Copy link
Collaborator

Thank you @yardencsGitHub.

We should consider adding an option for this to the config file and the ability to set it globally.
I think maybe we'd want a separate table for torch options like this.

[vak.torch]
float_32_matmul_precision = 'medium'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BUG Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants