enable/disable compile for quants methods #36519
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Thanks @SunMarc! This was on my mind as I ran into this last week :) |
Co-authored-by: Matthew Douglas <38992547+matthewdouglas@users.noreply.github.com>
Co-authored-by: Matthew Douglas <38992547+matthewdouglas@users.noreply.github.com>
src/transformers/generation/utils.py
Outdated
| generation_config.disable_compile = not self.hf_quantizer.is_compileable | ||
| else: | ||
| generation_config.disable_compile = False |
There was a problem hiding this comment.
nit:
generation_config here doesn't persist across calls, and, in general, I'm moving away from mutating generation_config so we can better distinguish user flags from our internal handling (especially in the decoding methods, like _sample).
Can we instead set is_compileable accordingly?
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
|
Let me know if this is better @gante ! |
What does this PR do ?
This PR disables compile if the quantization method do not support it. List to be found here. Need to add follow up PRs to allow compile or not for each method + add tests. Only torchao and bnb have the tests for now.
Fixes #36485 (comment)
cc @gante let me know what you think about that. I wanted to still have an easy way to not
disable_compileif we wanted.