[DEV Testing] Issues with `test_modeling_common`

[muellerzr]

I was writing some new tests to help us fix GA down in the modeling code and propagate the fix manually, since everything is breaking constantly with each fix and taking the dive.

When writing a test for this, which should be limiting to all ForCausalLM models, I was finding some strange errors on models that realistically shouldn't be trying to run these tests.

One of the general failures:

E       ValueError: Unrecognized configuration class <class 'transformers.models.vipllava.configuration_vipllava.VipLlavaConfig'> for this kind of AutoModel: AutoModelForCausalLM.
E       Model type should be one of AriaTextConfig

Said test:

    def test_training_gradient_accumulation(self):
        if not self.model_tester.is_training:
            self.skipTest(reason="ModelTester is not configured to run training tests")

        if len(self.all_generative_model_classes) == 0:
            self.skipTest(f"No generative model classes for {self.__class__.__name__}")

        for model_class in self.all_generative_model_classes:
            config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()

            with tempfile.TemporaryDirectory() as tmpdir:
                with torch.device(torch_device):
                    model_eager = AutoModelForCausalLM.from_config(
                        config, torch_dtype=torch.float32
                    )

                model_eager.save_pretrained(tmpdir)
                with torch.device(torch_device):
                    model = AutoModelForCausalLM.from_pretrained(
                        tmpdir, torch_dtype=torch.float32
                    )
                    inputs_dict["num_items_in_batch"] = inputs_dict["input_ids"].shape[0]
                    res = model(**inputs_dict, return_dict=False)[0]

List of tests:

FAILED tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.aria.configuration_aria.AriaConfig'> for this kind of AutoModel: AutoMod...
FAILED tests/models/autoformer/test_modeling_autoformer.py::AutoformerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.autoformer.configuration_autoformer.AutoformerConfig'> for this kind of ...
FAILED tests/models/bart/test_modeling_bart.py::BartModelTest::test_training_gradient_accumulation - TypeError: BartForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/bigbird_pegasus/test_modeling_bigbird_pegasus.py::BigBirdPegasusModelTest::test_training_gradient_accumulation - TypeError: BigBirdPegasusForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/blenderbot/test_modeling_blenderbot.py::BlenderbotModelTest::test_training_gradient_accumulation - TypeError: BlenderbotForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/blenderbot_small/test_modeling_blenderbot_small.py::BlenderbotSmallModelTest::test_training_gradient_accumulation - TypeError: BlenderbotSmallForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/blip_2/test_modeling_blip_2.py::Blip2ForConditionalGenerationDecoderOnlyTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.blip_2.configuration_blip_2.Blip2Config'> for this kind of AutoModel: Au...
FAILED tests/models/bloom/test_modeling_bloom.py::BloomModelTest::test_training_gradient_accumulation - ValueError: Got unexpected arguments: {'num_items_in_batch': 14}
FAILED tests/models/flava/test_modeling_flava.py::FlavaImageModelTest::test_training_gradient_accumulation - AttributeError: 'FlavaImageModelTester' object has no attribute 'is_training'
FAILED tests/models/flava/test_modeling_flava.py::FlavaMultimodalModelTest::test_training_gradient_accumulation - AttributeError: 'FlavaMultimodalModelTester' object has no attribute 'is_training'
FAILED tests/models/flava/test_modeling_flava.py::FlavaImageCodebookTest::test_training_gradient_accumulation - AttributeError: 'FlavaImageCodebookTester' object has no attribute 'is_training'
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.idefics.configuration_idefics.IdeficsConfig'> for this kind of AutoModel...
FAILED tests/models/idefics2/test_modeling_idefics2.py::Idefics2ForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.idefics2.configuration_idefics2.Idefics2Config'> for this kind of AutoMo...
FAILED tests/models/idefics3/test_modeling_idefics3.py::Idefics3ForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.idefics3.configuration_idefics3.Idefics3Config'> for this kind of AutoMo...
FAILED tests/models/imagegpt/test_modeling_imagegpt.py::ImageGPTModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.imagegpt.configuration_imagegpt.ImageGPTConfig'> for this kind of AutoMo...
FAILED tests/models/informer/test_modeling_informer.py::InformerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.informer.configuration_informer.InformerConfig'> for this kind of AutoMo...
FAILED tests/models/instructblip/test_modeling_instructblip.py::InstructBlipForConditionalGenerationDecoderOnlyTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.instructblip.configuration_instructblip.InstructBlipConfig'> for this ki...
FAILED tests/models/instructblipvideo/test_modeling_instructblipvideo.py::InstructBlipVideoForConditionalGenerationDecoderOnlyTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.instructblipvideo.configuration_instructblipvideo.InstructBlipVideoConfi...
FAILED tests/models/kosmos2/test_modeling_kosmos2.py::Kosmos2ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.kosmos2.configuration_kosmos2.Kosmos2Config'> for this kind of AutoModel...
FAILED tests/models/led/test_modeling_led.py::LEDModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.led.configuration_led.LEDConfig'> for this kind of AutoModel: AutoModelF...
FAILED tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava.configuration_llava.LlavaConfig'> for this kind of AutoModel: Auto...
FAILED tests/models/llava_next/test_modeling_llava_next.py::LlavaNextForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava_next.configuration_llava_next.LlavaNextConfig'> for this kind of A...
FAILED tests/models/llava_next_video/test_modeling_llava_next_video.py::LlavaNextVideoForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava_next_video.configuration_llava_next_video.LlavaNextVideoConfig'> f...
FAILED tests/models/llava_onevision/test_modeling_llava_onevision.py::LlavaOnevisionForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.llava_onevision.configuration_llava_onevision.LlavaOnevisionConfig'> for...
FAILED tests/models/longt5/test_modeling_longt5.py::LongT5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.longt5.configuration_longt5.LongT5Config'> for this kind of AutoModel: A...
FAILED tests/models/longt5/test_modeling_longt5.py::LongT5TGlobalModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.longt5.configuration_longt5.LongT5Config'> for this kind of AutoModel: A...
FAILED tests/models/m2m_100/test_modeling_m2m_100.py::M2M100ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.m2m_100.configuration_m2m_100.M2M100Config'> for this kind of AutoModel:...
FAILED tests/models/marian/test_modeling_marian.py::MarianModelTest::test_training_gradient_accumulation - TypeError: MarianForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/mbart/test_modeling_mbart.py::MBartModelTest::test_training_gradient_accumulation - TypeError: MBartForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/mllama/test_modeling_mllama.py::MllamaForCausalLMModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.mllama.configuration_mllama.MllamaTextConfig'> for this kind of AutoMode...
FAILED tests/models/mllama/test_modeling_mllama.py::MllamaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: The checkpoint you are trying to load has model type `mllama_text_model` but Transformers does not recognize this architecture. This ...
FAILED tests/models/moshi/test_modeling_moshi.py::MoshiTest::test_training_gradient_accumulation - TypeError: MoshiForCausalLM.forward() got an unexpected keyword argument 'moshi_audio_codes'
FAILED tests/models/mt5/test_modeling_mt5.py::MT5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.mt5.configuration_mt5.MT5Config'> for this kind of AutoModel: AutoModelF...
FAILED tests/models/mvp/test_modeling_mvp.py::MvpModelTest::test_training_gradient_accumulation - TypeError: MvpForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/nllb_moe/test_modeling_nllb_moe.py::NllbMoeModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.nllb_moe.configuration_nllb_moe.NllbMoeConfig'> for this kind of AutoMod...
FAILED tests/models/paligemma/test_modeling_paligemma.py::PaliGemmaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.paligemma.configuration_paligemma.PaliGemmaConfig'> for this kind of Aut...
FAILED tests/models/patchtsmixer/test_modeling_patchtsmixer.py::PatchTSMixerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.patchtsmixer.configuration_patchtsmixer.PatchTSMixerConfig'> for this ki...
FAILED tests/models/pegasus/test_modeling_pegasus.py::PegasusModelTest::test_training_gradient_accumulation - TypeError: PegasusForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.pegasus_x.configuration_pegasus_x.PegasusXConfig'> for this kind of Auto...
FAILED tests/models/pix2struct/test_modeling_pix2struct.py::Pix2StructModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.pix2struct.configuration_pix2struct.Pix2StructConfig'> for this kind of ...
FAILED tests/models/plbart/test_modeling_plbart.py::PLBartModelTest::test_training_gradient_accumulation - TypeError: PLBartForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/prophetnet/test_modeling_prophetnet.py::ProphetNetModelTest::test_training_gradient_accumulation - TypeError: ProphetNetForCausalLM.forward() got an unexpected keyword argument 'decoder_input_ids'
FAILED tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.qwen2_5_vl.configuration_qwen2_5_vl.Qwen2_5_VLConfig'> for this kind of ...
FAILED tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.qwen2_vl.configuration_qwen2_vl.Qwen2VLConfig'> for this kind of AutoMod...
FAILED tests/models/sam/test_modeling_sam.py::SamModelTest::test_training_gradient_accumulation - AttributeError: 'SamModelTester' object has no attribute 'is_training'
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithSpeechInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t.configuration_seamless_m4t.SeamlessM4TConfig'> for this kin...
FAILED tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t.configuration_seamless_m4t.SeamlessM4TConfig'> for this kin...
FAILED tests/models/seamless_m4t_v2/test_modeling_seamless_m4t_v2.py::SeamlessM4Tv2ModelWithSpeechInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t_v2.configuration_seamless_m4t_v2.SeamlessM4Tv2Config'> for ...
FAILED tests/models/seamless_m4t_v2/test_modeling_seamless_m4t_v2.py::SeamlessM4Tv2ModelWithTextInputTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.seamless_m4t_v2.configuration_seamless_m4t_v2.SeamlessM4Tv2Config'> for ...
FAILED tests/models/speech_to_text/test_modeling_speech_to_text.py::Speech2TextModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.speech_to_text.configuration_speech_to_text.Speech2TextConfig'> for this...
FAILED tests/models/superglue/test_modeling_superglue.py::SuperGlueModelTest::test_training_gradient_accumulation - AttributeError: 'SuperGlueModelTester' object has no attribute 'is_training'
FAILED tests/models/superpoint/test_modeling_superpoint.py::SuperPointModelTest::test_training_gradient_accumulation - AttributeError: 'SuperPointModelTester' object has no attribute 'is_training'
FAILED tests/models/switch_transformers/test_modeling_switch_transformers.py::SwitchTransformersModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.switch_transformers.configuration_switch_transformers.SwitchTransformers...
FAILED tests/models/t5/test_modeling_t5.py::T5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForC...
FAILED tests/models/time_series_transformer/test_modeling_time_series_transformer.py::TimeSeriesTransformerModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.time_series_transformer.configuration_time_series_transformer.TimeSeries...
FAILED tests/models/udop/test_modeling_udop.py::UdopModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.udop.configuration_udop.UdopConfig'> for this kind of AutoModel: AutoMod...
FAILED tests/models/umt5/test_modeling_umt5.py::UMT5ModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.umt5.configuration_umt5.UMT5Config'> for this kind of AutoModel: AutoMod...
FAILED tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.video_llava.configuration_video_llava.VideoLlavaConfig'> for this kind o...
FAILED tests/models/vipllava/test_modeling_vipllava.py::VipLlavaForConditionalGenerationModelTest::test_training_gradient_accumulation - ValueError: Unrecognized configuration class <class 'transformers.models.vipllava.configuration_vipllava.VipLlavaConfig'> for this kind of AutoMo...
FAILED tests/models/whisper/test_modeling_whisper.py::WhisperModelTest::test_training_gradient_accumulation - KeyError: 'input_ids'

cc @ydshieh @ArthurZucker

[ydshieh]

Hello @muellerzr . Do you already have a branch that you can share so we can run and check 🙏 ?

[muellerzr]

@ydshieh kinda. It's my own testing branch but basically add that test to test_modeling_common (my testing branch is about to get much worse for the bug I'm looking into 😓 )

[ydshieh]

OK, taking a look!

[ydshieh]

I think it's because

self.all_generative_model_classes doesn't only means AutoModelForCausalLM.

If restricting only to xxxForCausalLM models is an option, you can filter out within (at the beginning of ) for model_class in self.all_generative_model_classes: by checking against MODEL_FOR_CAUSAL_LM_MAPPING.

If you want to test for all kind of all_generative_model_classes, you can simply use model_class(....) I think.

[muellerzr]

@ydshieh that seems to somewhat work, but then leads me to:

______________________________________________ FlavaImageModelTest.test_training_gradient_accumulation _______________________________________________

self = <tests.models.flava.test_modeling_flava.FlavaImageModelTest testMethod=test_training_gradient_accumulation>

    def test_training_gradient_accumulation(self):
>       if not self.model_tester.is_training:
E       AttributeError: 'FlavaImageModelTester' object has no attribute 'is_training'

[muellerzr]

Which yes, I could just do:

        if not getattr(self.model_tester, "is_training", False):
            self.skipTest(reason="ModelTester is not configured to run training tests")

But wanted to point out another difference in existing code

[ydshieh]

Yeah, thanks. Not sure how

class FlavaImageModelTest(ModelTesterMixin)

is passing, maybe all tests that use self.model_tester.is_training for that model is skipped 😢

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.