[CODE] optimize BloomAttention, remove unnecessary host-to-device transfers

[justheuristic]

TL;DR the current code for BloomAttention is surprisingly inefficient, with obvious problems that should be easy to fix

The current code for BloomAttention is surprisingly inefficient:

• it does batch-repetition on CPU before copying larger tensor to GPU - https://github.com/huggingface/transformers/blob/ca2a55e9dfb245527b5e1c954fec6ffbb7aef07b/src/transformers/models/bloom/modeling_bloom.py#L358
• it checks for "0 in mask" on each layer, triggering needless CUDA synchronization https://github.com/huggingface/transformers/blob/ca2a55e9dfb245527b5e1c954fec6ffbb7aef07b/src/transformers/models/bloom/modeling_bloom.py#L361
• it checks runs a for-loop with synchronization to shift alibi weights inside each layer in each forward pass https://github.com/huggingface/transformers/blob/ca2a55e9dfb245527b5e1c954fec6ffbb7aef07b/src/transformers/models/bloom/modeling_bloom.py#L162-L167
• it computes alibi biases on CPU in python - https://github.com/huggingface/transformers/blob/ca2a55e9dfb245527b5e1c954fec6ffbb7aef07b/src/transformers/models/bloom/modeling_bloom.py#L94

It also misses a ton of opportunities for in-place ops for memory savings, but that's the least of its problems

[justheuristic]

Updates on alibi.py: https://gist.github.com/justheuristic/9751e02a2a5604a98a4fe0b6b688e808
Transformers PR developed by @younesbelkada : huggingface/transformers#17759

[justheuristic]

I've copy-pasted changes from @younesbelkada 's PR above.
As of right now ( aaaf0c2 ) , all four issues are solved.