Great work! Can I know if there is any implementation or script to call this model? Thanks.

[WilTay1]

No description provided.

[Edward-Sun]

Hi, thanks for your interest in our work. We are still working on code cleaning.

For now, you can install the LLaVA repo and use this gist to replace the model/builder.py file in llava.

Then, you can call the LLaVA-RLHF model by:

load_bf16 = True

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path, None, model_name, load_bf16=load_bf16)

model = PeftModel.from_pretrained(
      model,
      lora_path,
)

And remember to convert the input image to torch.bfloat16

images = images.to(dtype=torch.bfloat16)

[barshag]

So after replacing the gist, it will work straight-forward with the GRADIO? (Where should i write the code above?)