Assertion Error

[Whiskey-Bravo]

When i try to run the "docker compose up" command it downloads the model and then throws an AssertionError. I have tried deleting the model manually multiple times, but it still doesnt seem to work.

[1Wayne1]

same issues

[RichardScottOZ]

Yes, just tried this - same here -70b model as well.

[RichardScottOZ]

Model works in llama.cpp so perhaps this is a new config in the model issue?

[THeivers]

Seems to be a problem with the latest version of llama.cpp from what I can gather in https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML/discussions/14

I got it working locally (for now until the core problem is resolved) by following the suggestion to go back to an older version, so I replaced the image version with an older one.

For example

 llama-gpt-api-7b:
    image: ghcr.io/abetlen/llama-cpp-python:latest

to

llama-gpt-api-7b:
    image: ghcr.io/abetlen/llama-cpp-python@sha256:b6d21ff8c4d9baad65e1fa741a0f8c898d68735fff3f3cd777e3f0c6a1839dd4

This one was about 9 days ago (there were newer ones, but most were about a day old, so figured they might all share the same problem). So you could always try a later one than that. You can see all of them here: https://github.com/abetlen/llama-cpp-python/pkgs/container/llama-cpp-python/versions?filters%5Bversion_type%5D=untagged&page=1

[mayankchhabra]

Sorry for the confusion, folks! This was resolved as a part of #71 which added support for Code Llama models. It should be fixed in the master branch now. You were correct in your assessment @THeivers.

You can retry with:

git pull origin master
./run.sh --model 7b # or run this if you are on an M1/M2 mac: ./run-mac.sh --model 7b

Replace 7b with 13b, 70b, code-7b, code-13b, or code-34b.

[RichardScottOZ]

Thanks - llama.cpp is now GGUF by default and not GGML as I understand getting this working yesterday?