Cannot load pretrained model to run the inference s1_pipe

[aquarter147]

I encountered an issue while attempting to load a pre-trained model using the from_pretrained method in the transformers library. The error message indicates a discrepancy in the shape of the class_embedding tensor between the pre-trained model and the code. Specifically, the model expects a tensor of shape [1280] for class_embedding, while the code expects [1024]

CUDA_VISIBLE_DEVICES=0 python gen_s1.py --config cfgs/s1.yaml  validation_dataset.filepaths=['pumpkin.png'] validation_dataset.crop_size=224

/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
{'pretrained_model_name_or_path': '/home/thaoanh/workspace/Envision3D/ckpts/s1_pipe', 'revision': None, 'validation_dataset': {'root_dir': './example_imgs/', 'bg_color': 'white', 'crop_size': 224, 'img_wh': [256, 256], 'filepaths': ['pumpkin.png']}, 'seed': 43, 'validation_batch_size': 1, 'enable_xformers_memory_efficient_attention': True, 'dataloader_num_workers': 64, 'save_dir': './img_outputs/'}
Loading pipeline components...:  40%|████████████████████████████████████████████▊                                                                   | 2/5 [00:00<00:00, 27.49it/s]
Traceback (most recent call last):
  File "/home/thaoanh/workspace/Envision3D/gen_s1.py", line 230, in <module>
    main(cfg)
  File "/home/thaoanh/workspace/Envision3D/gen_s1.py", line 177, in main
    pipeline = load_envision3d_pipeline(cfg)
  File "/home/thaoanh/workspace/Envision3D/gen_s1.py", line 154, in load_envision3d_pipeline
    pipeline = MVDiffusionRefImagePipeline.from_pretrained(
  File "/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1265, in from_pretrained
    loaded_sub_model = load_sub_model(
  File "/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 520, in load_sub_model
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
  File "/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3677, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4104, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/transformers/modeling_utils.py", line 886, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/home/thaoanh/anaconda3/envs/wonder3dd/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 358, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([1280]) in "class_embedding" (which has shape torch.Size([1024])), this look incorrect
```.`

[Pang-Yatian]

Hi, we cannot reproduce this error on our side.

Please ensure the environment is consistent as listed in req.txt.