Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The specific settings of Lora finetuning #10

Open
Toneyaya opened this issue Apr 29, 2024 · 5 comments
Open

The specific settings of Lora finetuning #10

Toneyaya opened this issue Apr 29, 2024 · 5 comments

Comments

@Toneyaya
Copy link

Thank you for your open-source work!
Can you share the specific settings used for the LoRA Futuning results reported in the paper? I tried training with LoRA but the results differed from those reported in the paper (InstructBLIP).

@farewellthree
Copy link
Collaborator

Hi. When we conducted experiments using LoRA, we had not yet integrated DeepSpeed and transformer.trainer into the code. Therefore, we used stllm/train/train.py, with the following run configuration (model config and dataset config are the same):
run:
task: video_text_it

optimizer

lr_sched: "linear_warmup_cosine_lr"
init_lr: 1e-5
min_lr: 3e-6
warmup_lr: 4e-7

weight_decay: 0.05
max_epoch: 3
#iters_per_epoch: 200
batch_size_train: 1
batch_size_eval: 1
num_workers: 4
warmup_steps: 10000

seed: 42
output_dir: "path/to/output/dir"

amp: True
resume_ckpt_path: null

evaluate: False
train_splits: ["train"]

device: "cuda"
world_size: 1
dist_url: "env://"
distributed: True

@Toneyaya
Copy link
Author

Toneyaya commented May 1, 2024

Thank you for your reply! What are the settings of lora_r and lora_alpha?

@farewellthree
Copy link
Collaborator

64 and 16

@Toneyaya
Copy link
Author

Toneyaya commented May 8, 2024

The results of training using the mentioned configuration are currently even worse, with a score of 44 achieved on the first attempt, and performance on MVbench being less than 40. Here's the complete configuration file.
Additionally, several errors are encountered when attempting to train directly using train.py, such as the following ValueError: "Attempting to unscale FP16 gradients."
Could you kindly provide the code and configuration for LoRa fine-tuning? Your assistance would be greatly appreciated.

model:
  arch: st_llm_hf
  model_type: instructblip_vicuna0_btadapter
  use_grad_checkpoint: True
  max_txt_len: 256
  end_sym: "###"
  video_input: "all"
  llama_model: 'vicuna-7b-v1.1'
  ckpt: instruct_blip_vicuna7b_trimmed.pth'
  q_former_model: instruct_blip_vicuna7b_trimmed.pth'
  qformer_text_input: True
  freeze_LLM: True
  use_mask : True
  mvm_decode: True
  lora_r: 64
  lora_alpha: 16

datasets:
  caption_videochatgpt:
    num_frames: 16
    #video_reader_type: 'rawframe'
  classification_k710:
    num_frames: 16
  classification_ssv2:
    num_frames: 16
  reasoning_next_qa:
    num_frames: 16
  reasoning_clevrer_qa:
    num_frames: 16
  reasoning_clevrer_mc:
    num_frames: 16
  vqa_webvid_qa:
    num_frames: 16

run:
  task: video_text_it
  output_dir: "./stllm/output/instructblipbase_stllm_qa_lora_v1.1"
  lr_sched: "linear_warmup_cosine_lr"
  init_lr: 1e-5
  min_lr: 3e-6
  warmup_lr: 4e-7

  weight_decay: 0.05
  max_epoch: 3
  #iters_per_epoch: 200
  batch_size_train: 1
  batch_size_eval: 1
  num_workers: 4
  warmup_steps: 10000

  seed: 42
  amp: True
  resume_ckpt_path: null

  evaluate: False
  train_splits: ["train"]

  device: "cuda"
  world_size: 1
  dist_url: "env://"
  distributed: True

@farewellthree
Copy link
Collaborator

Sorry for the delayed response. I've been busy with the rebuttal lately. We will rerun the LoRA experiments and then provide you with the LoRA parameters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants