[BUG] torch._C._LinAlgError: linalg.cholesky always raised

[1649759610]

Describe the bug
Hi, @PanQiWei @TheBloke

Thanks for all contributions in quantization.

Recently, I try quantize a llama-style model with 16B params and 32k length，but unfortunately an Exception raised when quantize model at the 43-th layer. More detailed error msg is shown below .

I tried some solutions to solve it with more calibration examples and damp_percent:

• tried the number of calibration examples: 128, 256, 512, 1024
• tried damp_percent: 0.01, 0.1, 0.15, 0.2

But no matter what combination of the two params, torch._C._LinAlgError: linalg.cholesky is always raised when quantize model at the 43-th layer. More detailed error msg is shown below.

Any help would be appreciated.

Hardware details
GPU: A800
RAM: 400G

Software version
autogptq: 0.7.0+cu118
torch: 2.0.1
cuda: 11.8
python: 3.10

[Kk1984up]

I got the same error. have you solved the error now?

[1649759610]

Replacing calibration dataset seems to work sometime，but it is not the way that I wanna use.

[fxmarty]

Hi @1649759610 @Kk1984up seem to be the same issue: IST-DASLab/gptq#8 (comment)

Here's Elias suggestion: IST-DASLab/gptq#8 (comment)

[liuy-2019]

The same error.
But why calibration dataset can cause the Hessian Matrix not positive-definite?

[Qubitium]

Just ran into same issue. Semi-resolved in 636293f in Pr #640 : give user a hint to bypass error with ref to this issue