Consider using PyTorch 2.0 version of FlashAttention (remove dependency on flash-attn)

[Sciumo]

Per: Dao-AILab/flash-attention#203

if you're using pytorch 2.0 then FlashAttention is already available through torch.nn.functional.scaled_dot_product_attention.

The flash-attn project has build problems for many people.
Is it possible to consider using PyTorch 2.0 equivalent Flash Attention?

[vchiley]

For MPT we need to be able to use causal=True and we'd need to use attn_mask (aka attn_bias) to have ALiBi.

The variant exposed in scaled_dot_product_attention docs does not allow both. From docs:

[Sciumo]

So this issues needs to propagate to PyTorch.
Until then, I'll work with Flash Attn then as is.
Thanks.