TypeError: AutoAWQForCausalLM.from_quantized() got an unexpected keyword argument 'safetensors'

[flexchar]

I decided to try this instead of AutoGPTQ. I am trying this simplified snippet

pip install huggingface_hub transformers torch einops hf_transfer autoawq

from awq import AutoAWQForCausalLM
self.model = AutoAWQForCausalLM.from_quantized(
    MODEL_NAME, 
    fuse_layers=True,
    trust_remote_code=False, 
    safetensors=True
)

and I get the error

TypeError: AutoAWQForCausalLM.from_quantized() got an unexpected keyword argument 'safetensors' 

I'm using Modal.com service and nvidia/cuda:11.8.0-devel-ubuntu22.04 base image. Any ideas why I'd be getting this?

[casper-hansen]

Safetensors were only just implemented as they have not been released on PyPi yet. You must install from main to use this feature until the next release (hopefully soon).

[flexchar]

Did git clone https://github.com/casper-hansen/AutoAWQ && cd AutoAWQ && pip install ..

Now I'm greeted with
No module named 'awq.modules.fused

Runner failed with exception: ModuleNotFoundError("No module named 'awq.modules.fused'")
  File "/pkg/modal/_container_entrypoint.py", line 484, in call_function_async
    imp_fun.obj.__enter__()
  File "/root/pygmalion-13b-q4.py", line 72, in __enter__
    from awq import AutoAWQForCausalLM
  File "/usr/local/lib/python3.10/dist-packages/awq/__init__.py", line 2, in <module>
    from awq.models.auto import AutoAWQForCausalLM
  File "/usr/local/lib/python3.10/dist-packages/awq/models/__init__.py", line 1, in <module>
    from .mpt import MptAWQForCausalLM
  File "/usr/local/lib/python3.10/dist-packages/awq/models/mpt.py", line 70, in <module>
    from awq.modules.fused.block import MPTBlock
ModuleNotFoundError: No module named 'awq.modules.fused'

I'm inspired by https://huggingface.co/TheBloke/Pygmalion-2-13B-AWQ#install-the-necessary-packages

Will try rebuilding from source with -e parameter.

As far as I know, it was based on Llama 2 - not MPT model

EDIT:
Failed too

Cloning into 'AutoAWQ'...
Obtaining file:///AutoAWQ
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/AutoAWQ/setup.py", line 96, in <module>
          arch_flags = get_compute_capabilities()
        File "/AutoAWQ/setup.py", line 81, in get_compute_capabilities
          raise RuntimeError("GPUs with compute capability less than 8.0 are not supported.")
      RuntimeError: GPUs with compute capability less than 8.0 are not supported.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Terminating task due to error: failed to run builder command "git clone https://github.com/casper-hansen/AutoAWQ && cd AutoAWQ && pip install -e ."

I'll stash until you release 0.0.3 :)

[casper-hansen]

It looks like the GPU you are using is not compatible and then your installation fails. Hoping to release soon when #53 is merged

[casper-hansen]

I fixed the error below. The module was not found because it was missing the __init__.py file. Releasing v0.1.0 on PyPi today.

Runner failed with exception: ModuleNotFoundError("No module named 'awq.modules.fused'")
  File "/pkg/modal/_container_entrypoint.py", line 484, in call_function_async
    imp_fun.obj.__enter__()
  File "/root/pygmalion-13b-q4.py", line 72, in __enter__
    from awq import AutoAWQForCausalLM
  File "/usr/local/lib/python3.10/dist-packages/awq/__init__.py", line 2, in <module>
    from awq.models.auto import AutoAWQForCausalLM
  File "/usr/local/lib/python3.10/dist-packages/awq/models/__init__.py", line 1, in <module>
    from .mpt import MptAWQForCausalLM
  File "/usr/local/lib/python3.10/dist-packages/awq/models/mpt.py", line 70, in <module>
    from awq.modules.fused.block import MPTBlock
ModuleNotFoundError: No module named 'awq.modules.fused'

[casper-hansen]

pip install autoawq==0.1.0

[flexchar]

Got it working! Really excited to measure the speed improvements coming from AutoGPTQ. Thank you for getting this road block out of the way, Casper.