Exllama v2

[flozi00]

shell outputs:

root@71c94e4b85b3:/workspaces/lorax# curl 127.0.0.1:80/generate     -X POST     -d '{"inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]", "parameters": {"max_new_tokens": 64}}'     -H 'Content-Type: application/json'
{"generated_text":"\n\n> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia altogether in April and May?\n\n> Natalia sold clips to 48 of her friends in April, and then she"}

root@71c94e4b85b3:/workspaces/lorax# curl 127.0.0.1:80/generate     -X POST     -d '{"inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold halfps did Natalia sell altogether in April and May? [/INST]", "parameters": {"max_new_tokens": 64}}'     -H 'Content-Type: application/json'
{"generated_text":"\n\n> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia altogether in April and May?\n\n> Natalia sold clips to 48 of her friends in April, and then she"}

root@71c94e4b85b3:/workspaces/lorax# curl 127.0.0.1:80/generate     -X POST     -d '{"inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold halfps did Natalia sell altogether in April and May? [/INST]", "parameters": {"max_new_tokens": 64}}'     -H 'Content-Type: application/json'
{"generated_text":"\n\n> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia altogether in April and May?\n\n> Natalia sold clips to 48 of h

root@71c94e4b85b3:/workspaces/lorax# curl 127.0.0.1:80/generate     -X POST     -d '{"inputs": "### System: Du bist ein Chatbot Namens Egino. ### User: Wer bist du ? ### Assistant: ", "parameters": {"max_new_to
kens": 64}}'     -H 'Content-Type: application/json'
{"generated_text":"\n\nIch bin Egino, ein Chatbot, der Fragen beantwortet. \n\nWie kann ich dir heute helfen? "}

root@71c94e4b85b3:/workspaces/lorax# 

root@71c94e4b85b3:/workspaces/lorax# 

root@71c94e4b85b3:/workspaces/lorax# curl 127.0.0.1:80/generate     -X POST     -d '{"inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]", "parameters": {"max_new_tokens": 64, "adapter_id": "flozi00/Mistral-7B-german-assistant-v6"}}'     -H 'Content-Type: application/json'
{"generated_text":"\n\n> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia altogether in April and May?\n\n> Natalia sold clips to 48 of h

root@71c94e4b85b3:/workspaces/lorax# curl 127.0.0.1:80/generate     -X POST     -d '{"inputs": "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]", "parameters": {"max_new_tokens": 64, "adapter_id": "qblocks/mistral_7b_norobots"}}'     -H 'Content-Type: application/json'
{"generated_text":"\n\n> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia altogether in April and May?\n\n> Natalia sold clips to 48 of her friends in April, and then she"}

root@71c94e4b85b3:/workspaces/lorax# 

seems to be okay
Could you please also do more tests ?

[tgaddair]

LGTM! Just a nit and a question about deprecation for v1 kernels. I didn't review the CUDA kernels themselves, but I assume those are coming from the original exllama2 repo.

[tgaddair]

Since we're still building the v1 kernels, would it make sense to fallback to v1 if v2 is unavailable?

HAS_EXLLAMA_V2 = True
try:
    from lorax_server.utils.gptq.exllamav2 import QuantLinear as exllamav2QuantLinear
except ImportError:
    HAS_EXLLAMA_V2 = False

HAS_EXLLAMA_V1 = False
if not HAS_EXLLAMA_V2:
    HAS_EXLLAMA_V1 = True
    try:
        from lorax_server.utils.gptq.exllama import Ex4bitLinear
    except ImportError:
        HAS_EXLLAMA_V1 = False

...

if use_exllama:
    if HAS_EXLLAMA_V2:
            linear = exllamav2QuantLinear(qweight, qzeros, scales, g_idx, bias, bits, groupsize)
    else:
            linear = Ex4bitLinear(qweight, qzeros, scales, g_idx, bias, bits, groupsize)

Happy to defer to your judgement if you think it wouldn't make sense (in which case, we may wish to remove the v1 kernels from the repo and build process).

[flozi00]

As far I can see there is no problem with older architectures like flash v1 vs v2
Will take a deeper look, preferring to remove v1 to decrease docker size and maintainability

[tgaddair]

Hey @flozi00 just wanted to double-check if this was ready to merge before landing. Let me know :)

[flozi00]

From my Side it looks good
Maybe you could do another run to be sure everything is running correctly ?