Exception when saving weights from QLORA due to UnboundLocalError

[ethanhs]

System Info

• transformers version: 4.30.0.dev0
• Platform: Linux-6.1.0-9-amd64-x86_64-with-glibc2.36
• Python version: 3.10.10
• Huggingface_hub version: 0.15.1
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1+cu117 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@sgugger you reviewed the PR so it looks like your eyes were on this most recently. Relevant commit: 357f281

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run the following notebook: https://colab.research.google.com/github/utensil/llm-playground/blob/main/notebooks/axolotl/colab/axolotl_falcon_1b_qlora_gsm8k.ipynb.

I get the following error:

Traceback (most recent call last):
  File "/home/e/et/ethanhs/axolotl/axolotl/scripts/finetune.py", line 295, in <module>
    fire.Fire(train)
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/e/et/ethanhs/axolotl/axolotl/scripts/finetune.py", line 282, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 1661, in train
    return inner_training_loop(
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2070, in _inner_training_loop
    self._load_best_model()
  File "/home/e/et/ethanhs/miniconda3/lib/python3.10/site-packages/transformers/trainer.py", line 2258, in _load_best_model
    self._issue_warnings_after_load(load_result)
UnboundLocalError: local variable 'load_result' referenced before assignment

Expected behavior

I expect for transformers not to raise an exception.

[sgugger]

cc @younesbelkada

[younesbelkada]

Hi @ethanhs
Thanks for reporting #24103 should solve the issue

[ethanhs]

Thanks for the quick response and quick fix!