model-trainer-huggingface: llama2 7b isn't working

[samos123]

ValueError                                Traceback (most recent call last)                                                    
Cell In[6], line 15                                                                                                            
     12 model = prepare_model_for_kbit_training(model)                                                                         
     14 # add LoRA adaptor                                                                                                     
---> 15 model = get_peft_model(model, lora_config2)                                                                            
     16 model.print_trainable_parameters()                                                                                     
                                                                                                                               
File /usr/local/lib/python3.10/dist-packages/peft/mapping.py:98, in get_peft_model(model, peft_config, adapter_name)
     96 if isinstance(peft_config, PromptLearningConfig):                                                                      
     97     peft_config = _prepare_prompt_learning_config(peft_config, model_config)                                           
---> 98 return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name)
                                                                                                                               
File /usr/local/lib/python3.10/dist-packages/peft/peft_model.py:918, in PeftModelForCausalLM.__init__(self, model, peft_config,
 adapter_name)
    917 def __init__(self, model, peft_config: PeftConfig, adapter_name="default"):
--> 918     super().__init__(model, peft_config, adapter_name)
    919     self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation

File /usr/local/lib/python3.10/dist-packages/peft/peft_model.py:112, in PeftModel.__init__(self, model, peft_config, adapter_na
me)
    110 if not isinstance(peft_config, PromptLearningConfig):
    111     self.peft_config[adapter_name] = peft_config
--> 112     self.base_model = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type](
    113         self.base_model, self.peft_config, adapter_name
    114     )
    115     self.set_additional_trainable_modules(peft_config, adapter_name)


File /usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py:181, in LoraModel.__init__(self, model, config, adapter_name)
    179 self.forward = self.model.forward
    180 self.peft_config = config
--> 181 self.add_adapter(adapter_name, self.peft_config[adapter_name])
    183 # transformers models have a .config attribute, whose presence is assumed later on
    184 if not hasattr(self, "config"):

File /usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py:195, in LoraModel.add_adapter(self, adapter_name, config)
    193     config = self._prepare_lora_config(config, model_config)
    194     self.peft_config[adapter_name] = config
--> 195 self._find_and_replace(adapter_name)
    196 if len(self.peft_config) > 1 and self.peft_config[adapter_name].bias != "none":
    197     raise ValueError(
    198         "LoraModel supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters."
    199     )

File /usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py:357, in LoraModel._find_and_replace(self, adapter_name)
    354         self._replace_module(parent, target_name, new_module, target)
    356 if not is_target_modules_in_base_model:
--> 357     raise ValueError(
    358         f"Target modules {lora_config.target_modules} not found in the base model. "
    359         f"Please check the target modules and try again."
    360     )

ValueError: Target modules ['query_key_value'] not found in the base model. Please check the target modules and try again

[samos123]

Seem each Model will have different target modules, so need to come up with a way to detect the target modules automatically.

Reference: https://stackoverflow.com/questions/76736361/llama-qlora-error-target-modules-query-key-value-dense-dense-h-to-4h

[samos123]

Edit: a better approach would be to let peft automatically detect it: huggingface/peft#758 (comment)

Fallback if auto detection doesn't work:

Looks like by parsing config.json we should be able to dynamically detect the correct target modules: https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/pytorch_model.bin.index.json