[unrevert] Add batch invariant kernel override for FlashInfer backend [2/n] #26373
Conversation
Signed-off-by: Bram Wasti <bwasti@meta.com>
bwasti
force-pushed
the
det_flashinfer2
branch
from
297ecc2 to
fcfa198
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request reinstates support for batch invariant kernel overrides for the FlashInfer backend, which is a crucial feature for ensuring deterministic outputs regardless of batch composition. The changes primarily involve updating the FlashInfer dependency and plumbing through new parameters to enable a deterministic mode in the backend. Additionally, the batch invariance tests have been significantly improved to be more robust, configurable, and to cover the FlashInfer backend. My review focuses on the correctness and robustness of these changes. The implementation for enabling batch invariance in FlashInfer appears solid. I've identified a couple of minor issues in the updated test file where a hardcoded value should be replaced with a configurable variable to make the test more robust. Overall, the changes are well-implemented and the test enhancements are excellent.
Signed-off-by: Bram Wasti <bwasti@meta.com>
bwasti
force-pushed
the
det_flashinfer2
branch
from
fcfa198 to
5045121
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Bram Wasti <bwasti@fb.com>
… [2/n] (vllm-project#26373) Signed-off-by: Bram Wasti <bwasti@meta.com> Signed-off-by: Bram Wasti <bwasti@fb.com> Signed-off-by: 1994 <1994@users.noreply.github.com>
… [2/n] (vllm-project#26373) Signed-off-by: Bram Wasti <bwasti@meta.com> Signed-off-by: Bram Wasti <bwasti@fb.com> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>
… [2/n] (vllm-project#26373) Signed-off-by: Bram Wasti <bwasti@meta.com> Signed-off-by: Bram Wasti <bwasti@fb.com> Signed-off-by: bbartels <benjamin@bartels.dev>
2 participants
This change reinstates already approved + landed #25769 based on the latest bump to flashinfer:
#26326
It should not land before #26326