bug: inference of fine-tuning model is not working

[KayMKM]

I followed this document to fine-tune and deployed inference: https://github.com/microsoft/vscode-ai-toolkit/blob/main/doc/finetune.md

But I the inference endpoint occurs an error:

2025-01-09T06:46:43.857955385Z Traceback (most recent call last):
2025-01-09T06:46:43.857984860Z   File "/mount/inference/utils.py", line 55, in load_model
2025-01-09T06:46:43.875525329Z     model = AutoModelForCausalLM.from_pretrained(
2025-01-09T06:46:43.875556908Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-09T06:46:43.875562920Z   File "/opt/conda/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 559, in from_pretrained
2025-01-09T06:46:43.875625156Z     return model_class.from_pretrained(
2025-01-09T06:46:43.875632990Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-09T06:46:43.875660385Z   File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4000, in from_pretrained
2025-01-09T06:46:43.876266321Z     dispatch_model(model, **device_map_kwargs)
2025-01-09T06:46:43.876299283Z   File "/opt/conda/lib/python3.11/site-packages/accelerate/big_modeling.py", line 498, in dispatch_model
2025-01-09T06:46:43.876418477Z     model.to(device)
2025-01-09T06:46:43.876484914Z   File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2849, in to
2025-01-09T06:46:43.876775446Z     raise ValueError(
2025-01-09T06:46:43.876788781Z ValueError: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
2025-01-09T06:46:43.876792589Z 
2025-01-09T06:46:43.876795624Z During handling of the above exception, another exception occurred:
2025-01-09T06:46:43.876797928Z 
2025-01-09T06:46:43.876801214Z Traceback (most recent call last):
2025-01-09T06:46:43.876829868Z   File "/mount/inference/./gradio_chat.py", line 42, in <module>
2025-01-09T06:46:43.895541690Z     model = load_model(model_name, torch_dtype, quant_type)
2025-01-09T06:46:43.895572127Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-09T06:46:43.895577717Z   File "/mount/inference/utils.py", line 70, in load_model
2025-01-09T06:46:43.908149431Z     raise RuntimeError(f"Error loading model: {e}")
2025-01-09T06:46:43.908180659Z RuntimeError: Error loading model: `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.

I tried to upgrade the transformer package version to 4.47.1 but not working

[swatDong]

Could you please share more info like which model and what parameters?

[KayMKM]

Could you please share more info like which model and what parameters?

Here is the finetuning config:

{
  "SUBSCRIPTION_ID": "8f010c55-236f-461e-8dfa-3d05b41135dc",
  "RESOURCE_GROUP_NAME": "yuesu_fine_tuning_0109",
  "STORAGE_ACCOUNT_NAME": "aistoragecmatp",
  "FILE_SHARE_NAME": "aifilesharecmatp",
  "ACA_JOB_NAME": "aiacajobcmatp",
  "COMMANDS": [
    "cd /mount",
    "pip install huggingface-hub==0.22.2",
    "huggingface-cli download microsoft/Phi-3-mini-4k-instruct --revision main --local-dir ./model-cache/microsoft/Phi-3-mini-4k-instruct --local-dir-use-symlinks False --cache-dir ./cache/hfdownload",
    "pip install -r ./setup/requirements.txt",
    "python3 ./finetuning/invoke_olive.py && find models/ -print | grep adapter/adapter"
  ]
}

GPU is NC24-A100

[floatingsidewal]

I'm also running into the same issue, except when running locally (not on Azure subscription).

After a successful execution of invoke_olive.py, the examples such as console_chat.py fail:

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:46<00:00, 23.18s/it]
An error occurred: Can't find 'adapter_config.json' at '../models/qlora/qlora/gpu_model/adapter'

In the example I see, it looks like the call to from_pretrained in PEFT in inference/utils.py requires an adapters_name argument which should be a path to something like "../models/qlora/qlora/gpu_model/adapter"; however, no adapters seem to be present in the directory structure.

# utils.py
PeftModel.from_pretrained(model, adapters_name)

Models/qlora folder structure after a clean install and run:

[2/12/2025 1:30 PM]:models> ls -Recurse -Directory


    Directory: C:\Users\bwheel\git\LearnGPT\models


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         2/12/2025  11:21 AM                checkpoints
d-----         2/12/2025  11:32 AM                qlora


    Directory: C:\Users\bwheel\git\LearnGPT\models\checkpoints


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         2/12/2025  11:21 AM                checkpoint-2


    Directory: C:\Users\bwheel\git\LearnGPT\models\qlora


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         2/12/2025  11:32 AM                output_model


    Directory: C:\Users\bwheel\git\LearnGPT\models\qlora\output_model


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         2/12/2025  11:32 AM                model

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

[microsoft-github-policy-service]

Due to lack of details for further investigation, we will archive the issue for now. In case you still have following-up questions on this issue, please always feel free to reopen the issue by clicking ‘reopen issue’ button below the comment box. We will get back to you as soon as possible.