[Core] Allow AQLM on Pascal

[sasha0552]

AQLM works on Pascal.
AWQ generates gibberish (repeated exclamation mark !).
GPTQ is very slow. Even just loading Llama 3 8B takes 30 minutes. In comparison, FP16 takes 1 minute.
FP16 works fine.

AQLM

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  1042.37
Total input tokens:                      215196
Total generated tokens:                  182214
Request throughput (req/s):              0.96
Input token throughput (tok/s):          206.45
Output token throughput (tok/s):         174.81
---------------Time to First Token----------------
Mean TTFT (ms):                          295326.22
Median TTFT (ms):                        242159.76
P99 TTFT (ms):                           770025.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1470.49
Median TPOT (ms):                        1232.73
P99 TPOT (ms):                           7437.79
==================================================

AWQ

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  670.48
Total input tokens:                      215196
Total generated tokens:                  26482
Request throughput (req/s):              1.49
Input token throughput (tok/s):          320.96
Output token throughput (tok/s):         39.50
---------------Time to First Token----------------
Mean TTFT (ms):                          246708.10
Median TTFT (ms):                        228687.70
P99 TTFT (ms):                           577918.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6704.52
Median TPOT (ms):                        6226.30
P99 TPOT (ms):                           29068.24
==================================================

FP16

============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  756.78
Total input tokens:                      215196
Total generated tokens:                  187161
Request throughput (req/s):              1.32
Input token throughput (tok/s):          284.36
Output token throughput (tok/s):         247.31
---------------Time to First Token----------------
Mean TTFT (ms):                          236744.23
Median TTFT (ms):                        207316.46
P99 TTFT (ms):                           587200.91
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1161.41
Median TPOT (ms):                        877.68
P99 TPOT (ms):                           7518.22
==================================================

[qiudywzh]

Hi sasha0552, thanks for your great work!
I find that on pascal GPUs, AQLM inference throughput drops dramatically when the model getting bigger.
I could run Qwen1.5-32B-Chat-AQLM-1x16 on 1080Ti x 2 (which is amazing itself), but generate avg 3.24 token/s with avg 9.04s first token latency.