You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I can run the generate_unconditional_samples.py script on my GPU without issue, however, when I run the interactive_conditional_samples.py script, it crashes if there is more than one context token.
The interactive_conditional_samples.py script works fine as long as the model prompt only produces one context token, for instance using the prompt "please" produces the list of tokens [29688] and correctly generates text. However, it crashes if the model prompt produces two or more context tokens, for instance using the prompt "pig" produces the list of tokens [79, 328] and crashes immediately.
When it crashes I'm getting the error: failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Update
The problem occurs with tf1.12/1.15 or tf2.0, but disappear with tf2.3.0
I also meet this problem. After hours of debuging, I find it seems like a bug of tf. Suppose you input three token, the model.py will calculate below like logics in w = tf.matmul(q, k, transpose_b=True), which is OK during network initialization but will crash when execute session run.
I can run the generate_unconditional_samples.py script on my GPU without issue, however, when I run the interactive_conditional_samples.py script, it crashes if there is more than one context token.
The interactive_conditional_samples.py script works fine as long as the model prompt only produces one context token, for instance using the prompt "please" produces the list of tokens [29688] and correctly generates text. However, it crashes if the model prompt produces two or more context tokens, for instance using the prompt "pig" produces the list of tokens [79, 328] and crashes immediately.
When it crashes I'm getting the error:
failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
And a little further down I see:
If anyone has any insight on what might be going wrong, and how I can fix it, I'd really appreciate the help.
The text was updated successfully, but these errors were encountered: