TypeError: snapshot_download() got an unexpected keyword argument 'max_memory'

[upunaprosk]

An unexpected argument for snapshot_download when evaluating the model quantized with GPTQmodel.
Same issue as #1196

Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/lm_eval/__main__.py", line 459, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lm_eval/utils.py", line 458, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/lm_eval/evaluator.py", line 245, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lm_eval/api/model.py", line 155, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/lm_eval/models/huggingface.py", line 219, in __init__
    self._create_model(
  File "/lm_eval/models/huggingface.py", line 675, in _create_model
    self._model = GPTQModel.from_quantized(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/gptqmodel/models/auto.py", line 356, in from_quantized
    return MODEL_MAP[model_type].from_quantized(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/gptqmodel/models/loader.py", line 316, in from_quantized
    model_local_path = get_model_local_path(model_id_or_path, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/gptqmodel/models/loader.py", line 114, in get_model_local_path
    return snapshot_download(pretrained_model_id_or_path, **download_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
TypeError: snapshot_download() got an unexpected keyword argument 'max_memory'

Name: transformers
Version: 4.57.1
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: [transformers@huggingface.co](mailto:transformers@huggingface.co)
License: Apache 2.0 License
Location: /usr/local/lib/python3.12/dist-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: GPTQModel, lm_eval, peft, sentence-transformers, tokenicer
---
Name: GPTQModel
Version: 5.0.0
Summary: Production ready LLM model compression/quantization toolkit with hw accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.
Home-page: https://github.com/ModelCloud/GPTQModel
Author: 
Author-email: ModelCloud <[qubitium@modelcloud.ai](mailto:qubitium@modelcloud.ai)>
License: Apache License

[Qubitium]

@upunaprosk Can you provide the lm-eval cli command line or python script that triggered this?

Also you can try installing latest main brach. It will take like 10+ minutes due to kernel compilanation.

pip install -v . --no-build-isolation

[upunaprosk]

Here's the command

lm_eval --model hf --model_args pretrained=iproskurina/gemma-2-2b-gptqmodel-4bit,gptqmodel=True --device cuda:0 \ --tasks arc_easy \ --batch_size 8 --write_out --log_samples --output_path ./results-eval/ --trust_remote_code --seed 42

[upunaprosk]

for faster tests, this command:

lm_eval --model hf --model_args pretrained=iproskurina/opt-125m-int4-c4,gptqmodel=True --device cuda:0 --tasks arc_easy --batch_size 8

[upunaprosk]

pip install -v gptqmodel --no-build-isolation doesn't solve the issue for me

[Qubitium]

pip install -v gptqmodel --no-build-isolation doesn't solve the issue for me

Ok. But did you clone the main branch? You need to git clone the repo to get the latest code. pip install -v gptqmodel will not auto load latest unrelesed code.

git clone --depth 1 https://github.com/modelcloud/gptqmodel
cd gptqmodel
pip install -v . --no-build-isolation

[upunaprosk]

Yes, the first attempt for installing from source threw an error:

  ValueError: invalid pyproject.toml config: `project.license`.
  configuration error: `project.license` must be valid exactly by one definition (2 matches found):

      - keys:
          'file': {type: string}
        required: ['file']
      - keys:
          'text': {type: string}
        required: ['text']

  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 /usr/local/lib/python3.12/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpu77fwpto
  cwd: /content/gptqmodel
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

The reason was the wrong license format ; changing license format to license = { "text" = "MIT" } worked for me.

After correcting the license, I managed to install .
Currently testing again with lm-eval.

[upunaprosk]

license = { file = "licenses/LICENSE.apache" } works correctly for installation.
other minor issue I noticed during installation - 2 versions of pypcre in https://github.com/upunaprosk/GPTQModel/blob/main/pyproject.toml and reqs.

[Qubitium]

@upunaprosk License regression fixed in #2160

[upunaprosk]

@upunaprosk License regression fixed in #2160

same update can be done for dependencises, like logbar and pypcre?

[Qubitium]

@upunaprosk License regression fixed in #2160

same update can be done for dependencises, like logbar and pypcre?

Can you show me the errors for logbar/pypcre? I can't reproduce the setup license errors for those packages. gptqmodel had two locations which defined license-files so that was the problem here but those packages dont.

[upunaprosk]

@upunaprosk License regression fixed in #2160

same update can be done for dependencises, like logbar and pypcre?

Can you show me the errors for logbar/pypcre? I can't reproduce the setup license errors for those packages. gptqmodel had two locations which defined license-files so that was the problem here but those packages dont.

yes, I tried also in colab and face the same issue. here's the example:
https://colab.research.google.com/drive/1IOYl_2nyIhDeTGTqa4KhVNmxcN21HpBj?usp=sharing