[Usage] Error while using finetuned model

[yesgvinayak]

Describe the issue

Issue: I have fine tuned the llava-v1.5-7b. And in the output directory I got some files.

1. adapter_model.safetensors
2. config.json
3. README.md
4. adapter_config.json
5. non_lora_trainables.bin
6. trainer_state.json

then I tried inference using this folder as model_path and base model as liuhaotian/llava-v1.5-7b. I am getting error.

Command:

PASTE THE COMMANDS HERE.

Log:

RuntimeError: Error(s) in loading state_dict for LlavaLlamaForCausalLM:
        size mismatch for model.mm_projector.0.weight: copying a param with shape torch.Size([4096, 1024]) from checkpoint, the shape in current model is torch.Size([2097152, 1]).
        size mismatch for model.mm_projector.2.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).

Screenshots:
You may attach screenshots if it better explains the issue.

[itay1542]

It seems like you fine tuned using LoRa, which means you have to use the merge_lora_weights.py script (located under scripts/ folder) before trying inference.

[yesgvinayak]

@itay1542 I have ran the merge_lora_weights.py and created the merged one. but getting an error,

File "/home/skadmin/cx-research/core/Llava/llava/eval/run_llava.py", line 117, in eval_model
images_tensor = process_images(
File "/home/skadmin/cx-research/core/Llava/llava/mm_utils.py", line 171, in process_images
image = expand2square(image, tuple(int(x*255) for x in image_processor.image_mean))
AttributeError: 'NoneType' object has no attribute 'image_mean'

here is the code:
model_path = "./saved_model"
model_name = get_model_name_from_path(model_path)
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name)

[nlpkiddo-2001]

@itay1542 I have fine tuned the model without lora, Now what should i do for the inference. Kindly guide me