Benchmark vit xFormers

[bnabis93]

TODO

• Research about basic of xFormers
• Implement xFormers + ViT
• Benchmark xFormers + ViT (Each batchsize)

[bnabis93]

[Issue] Performance degradation in A100 GPU

• Vanilla Attention: 3.87ms
• Sparse Attention: 9.33ms
• Memory Efficient Attention: 6.34ms
• Sparse Attention is 2.4x slower than Vanilla Attention
• Memory Efficient Attention is 1.6x slower than Vanilla Attention

python reproduce.py

ViT Forward only
<torch.utils.benchmark.utils.common.Measurement object at 0x7fe7d59c9780>
profile
  Median: 3.87 ms
  IQR:    0.26 ms (3.76 to 4.02)
  514 measurements, 1 runs per measurement, 1 thread
Memory used: 28.87548828125 MB
Sparse ViT Forward only
<torch.utils.benchmark.utils.common.Measurement object at 0x7fe803ea64a0>
profile
  Median: 9.33 ms
  IQR:    0.21 ms (9.31 to 9.51)
  22 measurements, 10 runs per measurement, 1 thread
Memory used: 32.49267578125 MB
Mem efficient ViT Forward only
<torch.utils.benchmark.utils.common.Measurement object at 0x7fe803de7c10>
profile
  Median: 6.34 ms
  IQR:    0.08 ms (6.31 to 6.39)
  315 measurements, 1 runs per measurement, 1 thread
Memory used: 266.49072265625 MB
ViT average inference time : 3.186824321746826ms
ViT average inference time : 8.32324504852295ms
ViT average inference time : 5.165774822235107ms