video_miniGPT4

MiniGPT-4 for video

Currently, this is a simple extension of MiniGPT-4 without extra training. We try to undermine its ability for video understanding with simple prompt design.

🔥 Updates

• 2023/04/19: Simple extension release We simple encode 4 frames and use the time-sensitive prompt as follows:

      "First, <Img><ImageHere></Img>. Then, <Img><ImageHere></Img>. "
      "After that, <Img><ImageHere></Img>. Finally, <Img><ImageHere></Img>. "

  However, without video-text instruction finetuning, it's difficult to answer those questions about the time.

💬 Example

🏃 Usage

Please follow the instrction in MiniGPT-4 to prepare the environment.

• Prepare the envirment.

      conda env create -f environment.yml
      conda activate minigpt4

• Download BLIP2 model:
  • ViT: wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth
  • QFormer: wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained_flant5xxl.pth
  • Change the vit_model_path and q_former_model_path in minigpt4.yaml.
• Download Vicuna model:
  • LLAMA: Download it from the original repo or hugging face.
  • If you download LLAMA from the original repo, please process it via the following command:

      # convert_llama_weights_to_hf is copied from transformers
      python src/transformers/models/llama/convert_llama_weights_to_hf.py \
          --input_dir /path/to/downloaded/llama/weights \
          --model_size 7B --output_dir /output/path

  • Download Vicuna-13b-deelta-v0 and process it:

      # fastchat v0.1.10
      python3 -m fastchat.model.apply_delta \
      --base /path/to/llama-13b \
      --target /output/path/to/vicuna-13b \
      --delta lmsys/vicuna-13b-delta-v1.0

  • Change the llama_model in minigpt4.yaml.
• Download MiniGPT-4 model:
  • Linear layer can be downloaded here.
  • Change the ckpt in minigpt4_eval.yaml.
• Running demo:

      python demo_video.py --cfg-path eval_configs/minigpt4_eval.yaml

Acknowledgement

This project is mainly based on MiniGPT-4, which is support by Lavis, Vicuna and BLIP2. Thanks for these amazing projects!