Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integrated flash attention 2 (ver. 0.2.2 -> 2.2.1). Wall-clock performance improvement observed for 125M param model with seq length 4096 and batch size 128 running on 1 a10 GPU.
![image](https://private-user-images.githubusercontent.com/79075841/268773628-31e7ad77-9251-4839-9924-efaf78318a32.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkzNDY0NTYsIm5iZiI6MTcxOTM0NjE1NiwicGF0aCI6Ii83OTA3NTg0MS8yNjg3NzM2MjgtMzFlN2FkNzctOTI1MS00ODM5LTk5MjQtZWZhZjc4MzE4YTMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjI1VDIwMDkxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk5OWIzYWZlMjMzMjAwNDYwZTUxZDYwMzNlNjhmODg0Yjc1NmY0MzFmNTZmNDAxZWYwNjg3OTQ5YTdmZGU0MTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.3-Um-F9fNtqGVxsjn0G5DwZs3xgzJmXezaHcyvPR0I0)