Fix breaking change #1414

younesbelkada · 2024-01-30T09:25:01Z

Fix a breaking change in the recent release, I made a new PR as I messed up the commit history on the previous PR

pacman100

Thank you Younes! ⚡️

HuggingFaceDocBuilderDev · 2024-01-30T09:28:33Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

src/peft/utils/save_and_load.py

DreamGenX · 2024-02-04T16:31:28Z

Hey, looks like this fix is causing issues for me:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/py3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 59, in <module>
    fire.Fire(do_cli)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
    return do_train(parsed_cfg, parsed_cli_args)
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 55, in do_train
    return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
  File "/workspace/axolotl/src/axolotl/train.py", line 163, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1552, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1953, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2340, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2416, in _save_checkpoint
    self.save_model(staging_output_dir, _internal_call=True)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2927, in save_model
    self._save(output_dir)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2999, in _save
    self.model.save_pretrained(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/peft/peft_model.py", line 216, in save_pretrained
    output_state_dict = get_peft_model_state_dict(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/peft/utils/save_and_load.py", line 146, in get_peft_model_state_dict
    has_remote_config = file_exists(model_id, "config.json")
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2386, in file_exists
    get_hf_file_metadata(url, token=token)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1631, in get_hf_file_metadata
    r = _request_wrapper(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 385, in _request_wrapper
    response = _request_wrapper(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 408, in _request_wrapper
    response = get_session().request(method=method, url=url, **params)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 67, in send
    return super().send(request, *args, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: 303fade0-ec4c-44e0-b39a-0dacca0454aa)')
Traceback (most recent call last):
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/http/client.py", line 287, in _read_status

Traceback (most recent call last):
  File "/root/miniconda3/envs/py3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/py3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 59, in <module>
    fire.Fire(do_cli)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
    return do_train(parsed_cfg, parsed_cli_args)
  File "/workspace/axolotl/src/axolotl/cli/train.py", line 55, in do_train
    return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
  File "/workspace/axolotl/src/axolotl/train.py", line 163, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1552, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 1953, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2340, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2416, in _save_checkpoint
    self.save_model(staging_output_dir, _internal_call=True)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2927, in save_model
    self._save(output_dir)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2999, in _save
    self.model.save_pretrained(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/peft/peft_model.py", line 216, in save_pretrained
    output_state_dict = get_peft_model_state_dict(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/peft/utils/save_and_load.py", line 146, in get_peft_model_state_dict
    has_remote_config = file_exists(model_id, "config.json")
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2386, in file_exists
    get_hf_file_metadata(url, token=token)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1631, in get_hf_file_metadata
    r = _request_wrapper(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 385, in _request_wrapper
    response = _request_wrapper(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 408, in _request_wrapper
    response = get_session().request(method=method, url=url, **params)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 67, in send
    return super().send(request, *args, **kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: 303fade0-ec4c-44e0-b39a-0dacca0454aa)')
Traceback (most recent call last):
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/root/miniconda3/envs/py3.10/lib/python3.10/http/client.py", line 287, in _read_status

younesbelkada · 2024-02-04T16:37:17Z

Hmmm @DreamGenX it seems for some reason in your script HTTP requests are failing since file_exists will try to fetch from the web. Let me try to figure out a fix and get back here

DreamGenX · 2024-02-04T16:40:46Z

Moreover, I think there might be a bigger bug lurking here. If I am reading the code directly, when called with save_embedding_layers="auto" (which is the default and in many places can't be overwritten), then we will only save the embeddings if the number of tokens changes. But what if someone added embedding layers to modules_to_save, which unfreezes them regardless of whether tokens were added or not. Then the embeddings won't get saved?

DreamGenX · 2024-02-04T17:15:45Z

Hmmm @DreamGenX it seems for some reason in your script HTTP requests are failing since file_exists will try to fetch from the web. Let me try to figure out a fix and get back here

Just to add to this -- downloading the base model works fine, so it's strange that the file exists check would fail later on.

pacman100 · 2024-02-09T19:33:27Z

Hello @DreamGenX,

Please refer huggingface/trl#1287 (comment) for more context and how to explicitly set it to False when you don't need it.

But what if someone added embedding layers to modules_to_save, which unfreezes them regardless of whether tokens were added or not. Then the embeddings won't get saved

In that case also embedding layers would be saved.

* fix * Update src/peft/utils/save_and_load.py * Update src/peft/utils/save_and_load.py

fix

a54b2ca

pacman100 approved these changes Jan 30, 2024

View reviewed changes

younesbelkada commented Jan 30, 2024

View reviewed changes

src/peft/utils/save_and_load.py Outdated Show resolved Hide resolved

src/peft/utils/save_and_load.py Outdated Show resolved Hide resolved

younesbelkada added 2 commits January 30, 2024 10:29

Update src/peft/utils/save_and_load.py

f9bb7d3

Update src/peft/utils/save_and_load.py

06a55dc

younesbelkada merged commit 5eb5b49 into huggingface:main Jan 30, 2024
14 checks passed

younesbelkada deleted the fix-breaking-change branch January 30, 2024 09:46

younesbelkada mentioned this pull request Feb 13, 2024

[core / get_peft_state_dict] Ignore all exceptions to avoid unexpected errors #1458

Merged

BenjaminBossan pushed a commit to BenjaminBossan/peft that referenced this pull request Mar 14, 2024

Fix breaking change (huggingface#1414)

9437d74

* fix * Update src/peft/utils/save_and_load.py * Update src/peft/utils/save_and_load.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix breaking change #1414

Fix breaking change #1414

younesbelkada commented Jan 30, 2024

pacman100 left a comment

HuggingFaceDocBuilderDev commented Jan 30, 2024

DreamGenX commented Feb 4, 2024

younesbelkada commented Feb 4, 2024

DreamGenX commented Feb 4, 2024

DreamGenX commented Feb 4, 2024

pacman100 commented Feb 9, 2024

Fix breaking change #1414

Fix breaking change #1414

Conversation

younesbelkada commented Jan 30, 2024

pacman100 left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jan 30, 2024

DreamGenX commented Feb 4, 2024

younesbelkada commented Feb 4, 2024

DreamGenX commented Feb 4, 2024

DreamGenX commented Feb 4, 2024

pacman100 commented Feb 9, 2024