ptxas returned an error during compilation of ptx to sass

[2subzero]

When running alphafold on Centos7, encountering ptxas error

I0811 21:45:02.256136 47363873189568 templates.py:270] Found an exact template match 3fwt_A. I0811 21:45:04.627611 47363873189568 run_alphafold.py:142] Running model model_1 I0811 21:46:29.244786 47363873189568 model.py:131] Running predict with shape(feat) = {'aatype': (4, 560), 'residue_index': (4, 560), 'seq_length': (4,), 'template_aatype': (4, 4, 560), 'template_all_atom_masks': (4, 4, 560, 37), 'template_all_atom_positions': (4, 4, 560, 37, 3), 'template_sum_probs': (4, 4, 1), 'is_distillation': (4,), 'seq_mask': (4, 560), 'msa_mask': (4, 508, 560), 'msa_row_mask': (4, 508), 'random_crop_to_size_seed': (4, 2), 'template_mask': (4, 4), 'template_pseudo_beta': (4, 4, 560, 3), 'template_pseudo_beta_mask': (4, 4, 560), 'atom14_atom_exists': (4, 560, 14), 'residx_atom14_to_atom37': (4, 560, 14), 'residx_atom37_to_atom14': (4, 560, 37), 'atom37_atom_exists': (4, 560, 37), 'extra_msa': (4, 5120, 560), 'extra_msa_mask': (4, 5120, 560), 'extra_msa_row_mask': (4, 5120), 'bert_mask': (4, 508, 560), 'true_msa': (4, 508, 560), 'extra_has_deletion': (4, 5120, 560), 'extra_deletion_value': (4, 5120, 560), 'msa_feat': (4, 508, 560, 49), 'target_feat': (4, 560, 22)} 2021-08-11 21:46:29.891255: W external/org_tensorflow/tensorflow/stream_executor/gpu/asm_compiler.cc:81] Couldn't get ptxas version string: Internal: Running ptxas --version returned 32512 2021-08-11 21:46:29.922824: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:479] ptxas returned an error during compilation of ptx to sass: 'Internal: ptxas exited with non-zero error code 32512, output: ' If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided. Fatal Python error: Aborted

[Augustin-Zidek]

Have you checked whether you have enough disk space as the error mentions?

If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.

[Ftamsir]

Finally find same problem with mine. I'm running on Ubuntu2004 with 3080Ti cuda 11.4 cudnn8.2.1 and i see the ptxas error too. I tried other cuda and cudnn version like 11.1 + 8.1.1 but it didn't work. Otherwise cudnn samples and tensorflow.test shows my gpu is working. I guess it's the jax or jaxlib problem. Hope you make progress

[2subzero]

Have you checked whether you have enough disk space as the error mentions?

If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.

I checked, and there is enough space.

[benfulton]

I got this error when ptxas wasn't in my path. I added it to the path and the error went away.

[Ftamsir]

I got this error when ptxas wasn't in my path. I added it to the path and the error went away.

i used run_docker.py and i set
PATH=/usr/local/cuda-11.4/bin:$PATH
but it still got
I0901 12:14:38.792916 140640899000128 run_docker.py:193] 2021-09-01 04:14:38.791988: W external/org_tensorflow/tensorflow/stream_executor/gpu/asm_compiler.cc:81] Couldn't get ptxas version string: Internal: Couldn't invoke ptxas --version I0901 12:14:38.794684 140640899000128 run_docker.py:193] 2021-09-01 04:14:38.793811: F external/org_tensorflow/tensorflow/compiler/xla/service/gpu/nvptx_compiler.cc:479] ptxas returned an error during compilation of ptx to sass: 'Internal: Failed to launch ptxas' If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided. I0901 12:14:38.795017 140640899000128 run_docker.py:193] Fatal Python error: Aborted
can you please show me how you added it to the path? thank you anyway

[Lsz-20]

the same question,have you solved this ？@2subzero
waiting for your help! @Ftamsir @benfulton

[Augustin-Zidek]

This should be fixed by bumping the CUDA version in AlphaFold v2.1.2. Feel free to reopen if that is not the case.

[hn42]

thenks @benfulton

add ptxas filepath
or
add ptxas file in project (I solved like this)

[asmit203]

thenks @benfulton

add ptxas filepath or add ptxas file in project (I solved like this)

Can you please tell me how did you add then ptxas to the project.