Fix regression compressed-tensors

[SunMarc]

What does this PR do?

This PR fixes the compressed-tensors model loading as it was broken for some models by this PR. Impacted models are models with config attribute "quantization_status": "frozen" (e.g. this is the case for the FP8 model) because we defaulted run_compressed to be True in the config. Thus, it triggers an error immediately.

Fixes #36915

To reproduce:

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("neuralmagic/Meta-Llama-3.1-8B-FP8")
model = AutoModelForCausalLM.from_pretrained("neuralmagic/Meta-Llama-3.1-8B-FP8")

Also maybe we shouldn't run the model with run_compressed in the case the model is frozen, but for the fp8 model, the weights are actually in fp8 format.

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

cc @rahul-tuli

[dsikka]

This is not always True.

The model listed is an old model which is failing as the config itself is incorrect and should be updated or run_compressed should be passed in as False.

The FROZEN state is reserved for models that have already been decompressed.

[SunMarc]

Sounds good then, this is what I thought . Can you update the config for those models (the most used ones should be enough !) ?

[SunMarc]

LMK when this is done and I will close this PR !

[rahul-tuli]

Hi @SunMarc I have updated our most downloaded models, you should be good to go!

[SunMarc]

Thanks !