Florence2 stopped working after upgrade to 4.50.0 ("Unrecognized configuration class")

[senarvi]

System Info

The moment the latest transformers version 4.50.0 appeared in PyPI, Florence2 stopped working. I'm new to Huggingface and I'm following the "how to get started" example. All I do is

AutoModelForCausalLM.from_pretrained(
    "microsoft/Florence-2-large", torch_dtype=torch.float16, trust_remote_code=True
)

This works with 4.49.0, but with 4.50.0 I get

Traceback (most recent call last):
  File "/florence2/detect.py", line 15, in <module>
    AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch.float16, trust_remote_code=True)
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 576, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.microsoft.Florence-2-large.00d2f1570b00c6dea5df998f5635db96840436bc.configuration_florence2.Florence2LanguageConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of AriaTextConfig, BambaConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, Cohere2Config, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, DiffLlamaConfig, ElectraConfig, Emu3Config, ErnieConfig, FalconConfig, FalconMambaConfig, FuyuConfig, GemmaConfig, Gemma2Config, Gemma3Config, Gemma3TextConfig, GitConfig, GlmConfig, GotOcr2Config, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GraniteConfig, GraniteMoeConfig, GraniteMoeSharedConfig, HeliumConfig, JambaConfig, JetMoeConfig, LlamaConfig, MambaConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MllamaConfig, MoshiConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, OlmoConfig, Olmo2Config, OlmoeConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, PhimoeConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, ZambaConfig, Zamba2Config.

I tried to check what makes the difference. The reason for the error is that in _BaseAutoModelClass.from_pretrained(), has_remote_code should evaluate to True, because if I print the config, it looks like this:

Florence2Config {
  "_name_or_path": "microsoft/Florence-2-large",
  "architectures": [
    "Florence2ForConditionalGeneration"
  ],
  "auto_map": {
    "AutoConfig": "microsoft/Florence-2-large--configuration_florence2.Florence2Config",
    "AutoModelForCausalLM": "microsoft/Florence-2-large--modeling_florence2.Florence2ForConditionalGeneration"
  },
  ...

But after the version upgrade, has_remote_code evaluates to False, even though the config, looks like this:

Florence2Config {
  "architectures": [
    "Florence2ForConditionalGeneration"
  ],
  "auto_map": {
    "AutoConfig": "microsoft/Florence-2-large--configuration_florence2.Florence2Config",
    "AutoModelForCausalLM": "microsoft/Florence-2-large--modeling_florence2.Florence2ForConditionalGeneration"
  },
...

Who can help?

@amyeroberts @qubvel

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Install transformers==4.50.0
2. Run

AutoModelForCausalLM.from_pretrained(
    "microsoft/Florence-2-large", torch_dtype=torch.float16, trust_remote_code=True
)

Expected behavior

This should not crash.

[qubvel]

@Rocketknight1 do you have an idea of what has been changed in remote code with the recent release?

[zucchini-nlp]

cc @gante i believe this is related to our changes in CausalLM 🙃 We forgot about custom code users

I can take a detailed look during weekend, just flagging it for now :)

[mdkberry]

again. I only just fixed this yday on my machine by pip uninstalling transformers from comfyui python_embed folder, and running Kijai version of the Florence2 nodes again which installed a different transformers version, 49 I think. I think its a clash with transformer version that is getting installed.

but I just updated comfyui to nightly this morning, and its happened again. I get - "Unrecognized configuration class <class 'transformers_modules.Florence-2-large.configuration_florence2.Florence2LanguageConfig'> for this kind of AutoModel: AutoModelForCausalLM."

will test the fix and report back as it might not be working any more.

[elkay]

If you don't want to go through all sorts of hackery while waiting for a fix, you can just back out to transformers 4.49.0 for the moment.

From your python environment:

python -m pip install --upgrade transformers==4.49.0

When the fix comes out you can just upgrade to the latest version running the command again but removing the version number.