Add `optimum.quanto` as supported load-time `quantization_config`

[vladmandic]

Recent additions to diffusers added BitsAndBytesConfig as well as TorchAoConfig options that can be used as quantization_config when loading model components using from_pretrained

for example:

quantization_config = BitsAndBytesConfig(...)
transformer = SD3Transformer2DModel.from_pretrained(repo_id, subfolder="transformer", quantization_config=quantization_config)

ask is to also support Huggingface's own Optimum Quanto
right now its possible to use it, but only as post-load on-demand quantization, there is no option to use it like BnB or TorchAO to apply quantization automatically during load itself.

@yiyixuxu @sayakpaul @DN6 @asomoza

[sayakpaul]

This should be relatively easy to add. @DN6 @a-r-r-o-w any of you would like to take it up? Since we have three new backends and a guide to add a new backend for quantization, I think this could be opened for community contributions, too.

[a-r-r-o-w]

I have a few things on my plate on the optimization and performance side that I would like to get out first in the next 1-2 weeks. Will be happy to look into it if not picked up by then 🔜

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[vladmandic]

ping to remove stale

[vladmandic]

any update on this one? its been marked as wip for 1.5 months without progress....

[sayakpaul]

#10756

1.5 months minus a few days.

[vladmandic]

#10756

1.5 months minus a few days.

aaa - great! didn't see it since it was not linked to this request.

[DN6]

Closing as completed since #10756 was merged.