Add fix for CPU Inference

[vivekkhandelwal1]

Signed-Off By: Vivek Khandelwal vivek@nod-labs.com

[vivekkhandelwal1]

@fxmarty, need your review on this.

[younesbelkada]

Thanks! I think you don't need to hardcode the dtype - wdyt?

[fxmarty]

A safe approach would be to modify https://github.com/PanQiWei/AutoGPTQ/blob/518617b8d682aaa95796f622d788e014ee882869/auto_gptq/modeling/_utils.py#L70 to pass the dtype to QuantLinear init, and to default to fp16 (ugly, but for backward compatibility).

This may not work for transformers integration though - maybe torch.get_default_dtype() would do.

[vivekkhandelwal1]

@fxmarty, can you please review it now? I think we need to make the changes here https://github.com/huggingface/optimum/blob/8e7588b09df2f15c47e9b92f81ec2b05f7ae6957/optimum/gptq/quantizer.py#L242-L246 as well?

[vivekkhandelwal1]

I have added a PR here: huggingface/optimum#1496

[fxmarty]

LGTM, can you make sure the tests pass?

CUDA_VISIBLE_DEVICES=0 pytest tests/ -s -vvvvv

[vivekkhandelwal1]

@fxmarty I have updated the PR with the required changes.

[fxmarty]

LGTM!

Could you confirm that tests pass?

[vivekkhandelwal1]

CUDA_VISIBLE_DEVICES=0 pytest tests/ -s -vvvvv

The following 2 tests failed:

FAILED tests/test_q4.py::TestsQ4CUDA::test_cuda_old_0 - AssertionError: False is not true
FAILED tests/test_q4.py::TestsQ4CUDA::test_cuda_old_1 - AssertionError: False is not true

Edit:

_________________________________________________________________________________ TestsQ4CUDA.test_cuda_old_0 _________________________________________________________________________________

a = (<tests.test_q4.TestsQ4CUDA testMethod=test_cuda_old_0>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../shark-vivekkhandelwal1/shark.venv/lib/python3.11/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_q4.py:356: in test_cuda_old
    self.assertTrue(linear.autogptq_cuda_available)
E   AssertionError: False is not true
_________________________________________________________________________________ TestsQ4CUDA.test_cuda_old_1 _________________________________________________________________________________

a = (<tests.test_q4.TestsQ4CUDA testMethod=test_cuda_old_1>,), kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

../shark-vivekkhandelwal1/shark.venv/lib/python3.11/site-packages/parameterized/parameterized.py:620: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_q4.py:356: in test_cuda_old
    self.assertTrue(linear.autogptq_cuda_available)
E   AssertionError: False is not true

[vivekkhandelwal1]

The above failure doesn't seem to be occurring because of the changes made in this PR. @fxmarty

[fxmarty]

Yes it was fixed by #387. Thank you!

[vivekkhandelwal1]

Thanks, @fxmarty, for your support in getting these patches in.

[chuyqa]

This seems broken in master at the moment:

File "/opt/miniconda3/envs/text-gen-gptq/lib/python3.10/site-packages/auto_gptq/nn_modules/qlinear/qlinear_cuda.py", line 272, in forward
out = out.to(dtype=weights.dtype).reshape(out_shape)
UnboundLocalError: local variable 'weights' referenced before assignment

[vivekkhandelwal1]

Hi @chuyqa, thanks for pointing out this issue. I have added a fix here: #390.