-
Notifications
You must be signed in to change notification settings - Fork 128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed to find best cuBLAS algorithm, GEMM performance might be suboptimal #62
Comments
Can you share the output of |
These are the outputs:
|
Thanks Yoshitaka, I'll look into those issues and I'll come back.
Best, Fernando
…On Thu, Feb 24, 2022 at 8:39 PM Yoshitaka Moriwaki ***@***.***> wrote:
11098MiB / 11441MiB
The Memory-usage seems very high. Is any other process using GPU running?
Or, are these issues #22
<#22>, #27
<#27>, and #50
<#50> helpful to you?
—
Reply to this email directly, view it on GitHub
<#62 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AK537YQCJJOEMHZKBJSLFKLU426TXANCNFSM5PFHBHTQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi Yoshitaka, Even though the same warnings were showing in the terminal, I was surprised to see that now the GPU was being used almost at capacity (GPU's memory was also >97% used). However, after a while, the process stopped with the following message:
The complete terminal's output in the text file. Thanks, |
Hello, I have an update for this. Today I ran the same test, but this time WITHOUT the commands indicated in the issues:
Now, the calculations were completed, with the GPU being used at capacity and taking one hour for a ~430 residues model. Still, the Thanks, |
Dear Yoshitaka,
I was able to successfully install ColabFold in a machine runnine Ubuntu 20 and a Tesla K40c GPU. Running a prediction is completed, although it takes ~24 hs to be completed (430 residues long), with command
colabfold_batch --amber --templates --num-recycle 3 test.fasta /home/pc08/Desktop/test_AF
During calculation, I get the following messages:
E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:771] failed to alloc 23994040320 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
W external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gemm_algorithm_picker.cc:211] Failed to find best cuBLAS algorithm, GEMM performance might be suboptimal: INTERNAL: All algorithms tried for %cublas-gemm.21 = f32[1908,128]{1,0} custom-call(f32[1908,128]{1,0} %bitcast.319, f32[128,128]{1,0} %bitcast.321), custom_call_target="__cublas$gemm", backend_config="{\"alpha_real\":1,\"alpha_imag\":0,\"beta\":0,\"dot_dimension_numbers\":{\"lhs_contracting_dimensions\":[\"1\"],\"rhs_contracting_dimensions\":[\"0\"],\"lhs_batch_dimensions\":[],\"rhs_batch_dimensions\":[]},\"batch_size\":\"1\",\"lhs_stride\":\"244224\",\"rhs_stride\":\"16384\"}" failed. Falling back to default algorithm.
Let me repeat, that nevertheless, the calculations of the 5 models are completed, and looking good. But I would like to know if this can be improved to run it more efficiently.
Not sure if it's useful, but I've noticed that the GPU is only sporadically used (with GPU's memory used to the max the whole time), and one CPU being used at 100% during most of the run.
If you have any insights on how to resolve this, I would greatly appreciate it. Thanks!
The text was updated successfully, but these errors were encountered: