Cuda Dockerfile - model produces garbage output on GPU (if layers offloaded to GPU)

[pradhyumna85]

Current Behavior

If we use the cuda dockerfile (https://github.com/abetlen/llama-cpp-python/blob/main/docker/cuda_simple/Dockerfile) and use it with any ggml model file and offloading layers to gpu, model outputs garbage text.

By garbage I mean, absolute garbage - symbols, special characters, Unicode characters etc, no English or any language.

Environment and Context

Running on centos 7 with docker and nvidia docker toolkit installed
Instance - aws g4dn.12xlarge (4 Nvidia T4 - 64GB VRAM)

Exact same issue seen in a repo related to Cublas in koboldcpp which also kind of uses llamacpp

issue: bartowski1182/koboldcpp-docker#1
fix commit: bartowski1182/koboldcpp-docker@331326a

Fix

I'll raise a pull request, with the updated dockerfile.

Cheers !🥂

[pradhyumna85]

Experienced similar issue, it returns stuff like content": "The\u0015\u0016\u0013\t\u0018!\u001d\u0019\u000e\u001f\u0005\u001e\u0011 ". Which is obviously too small for any normal utf-8 chars.

@afafw, The only meaningful word in your output - "The" probably came due to not all layers offloaded to GPU(s), am I correct?

[afafw]

Actually, that wasn't the case, I loaded all 35 layers into it. But I understand what you mean, since I did tried to load only part of network into gpu and it wasn't completely rubbish. It was just generating weird words.

[afafw]

I think the key is also "ENV CUDA_DOCKER_ARCH=all" as I just saw your pull request.

[pradhyumna85]

BTW, you can try this dockerfile. It worked for me.
Dockerfile.txt

@afafw, yeah I already fixed it, also raised a PR with the issue itself. You can check my PR, which is much shorter and cleaner.
Thank you anyway.

[afafw]

Yeah, I just noticed it, I probably deleted this line

BTW, you can try this dockerfile. It worked for me.
Dockerfile.txt

@afafw, yeah I already fixed it, also raised a PR with the issue itself. You can check my PR, which is much shorter and cleaner. Thank you anyway.

I saw it just after I send this, I thought I probably deleted this. I think your dockerfile probably works. As mine was simply building from source.

[pradhyumna85]

Fixed with the merge of PR #598