How to disable flash_attention

[lqniunjunlper]

When i run test code, got the error:
RuntimeError: FlashAttention only supports Ampere GPUs or newer.

[Sunwood-ai-labs]

I also encounter the same problem

[OmkarThawakar]

Hi @lqniunjunlper and @Sunwood-ai-labs ,

Thanks for the interest in our work.

In place of flash_attention you can use default PyTorch attention.

Just replace line "from flash_attn import flash_attn_func" with "torch.nn.functional.scaled_dot_product_attention" .

[lqniunjunlper]

@OmkarThawakar Thanks, it's worked!
But i got another error like this "
attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: The size of tensor a (10) must match the size of tensor b (19) at non-singleton dimension 1 "
May the shape of q, k,v not matched!

[simplew2011]

same

[Luoyingfeng8]

@OmkarThawakar Thanks, it's worked! But i got another error like this " attn_output = torch.nn.functional.scaled_dot_product_attention( RuntimeError: The size of tensor a (10) must match the size of tensor b (19) at non-singleton dimension 1 " May the shape of q, k,v not matched!

Do you solve the problem?

[zimenglan-sysu-512]

+1, @OmkarThawakar did u solve it?
thanks