Table of Best Results

[SilvaRaulEnrique]

The idea of this post is construct and maintain colaborativaly a complete and actualizated best orders ranking list table to compile and executate, considering best models results for the more very long context with efectively complety coherent response, by example:

For NVidia 3090 (with CUDA and 24 GB RAM), for now the best orders are:

compile:
make LLAMA_CUBLAS=1

run main example:

./main \
-m models/TheBloke/llama-2-13b-chat.ggmlv3.q8_0.bin \
-ngl 254 \
-c 4096 \
-gqa 8 \
--reverse-prompt "User:" \
--file chat_dies_User_Assistant.txt \
--in-prefix ' ' "$@"

or?

./main \
-m models/s3nh/longchat-7b-v1.5-32k.ggmlv3.q8_0.bin \
-ngl 254 \
-c 32768 \
--rope-scale 8 \
--reverse-prompt "User:" \
--file chat_dies_User_Assistant.txt \
--in-prefix ' ' "$@"

fine tuning? train-text-from scratch:

./train-text-from-scratch \
--vocab-model ServidorIA/models/models/TheBloke/llama-2-13b-chat.ggmlv3.q8_0.bin \
--ctx 64 \
--embd 256 \
--head 8 \
--layer 16 \
--checkpoint-out chk-Ultimas_Acordadas_y_Circulares-256x16.bin \
--model-out ggml-Ultimas_Acordadas_y_Circulares-256x16-f32.bin \
--train-data "Ultimas_Acordadas_y_Circulares.txt" \
-t 4 \
-b 8 \
-n 32 \
--seed 1 \
--adam-iter 16 \
--print-details-interval 0 \
--predict 16 \
--use-flash \
--mem-compute 8

or?

./train-text-from-scratch
--vocab-model ServidorIA/models/s3nh/longchat-7b-v1.5-32k.ggmlv3.q8_0.bin
--ctx 64
--embd 256
--head 8
--layer 16
--checkpoint-out chk-Ultimas_Acordadas_y_Circulares-256x16.bin
--model-out ggml-Ultimas_Acordadas_y_Circulares-256x16-f32.bin
--train-data "Ultimas_Acordadas_y_Circulares.txt"
-t 4
-b 8
-n 32
--seed 1
--adam-iter 16
--print-details-interval 0
--predict 16
--use-flash
--mem-compute 8

Note: models TheBloke, and s3nh are in https://huggingface.co/

¿There is the best parameters or considerer "now" one best?, ¿and for another hardware configuration?

[JRZS]

Running on an M2, built from scratch, I get this error:
error: unknown argument: -gqa

Anyone know why this is happening?

[ScarletEmerald]

-gqa used to be needed when loading Llama2 70B model, but in the current code with the new GGUF format it is no longer used.

[JRZS]

The problem is when you run it now, you get the pre-gqa error:
error loading model: create_tensor: tensor 'blk.0.attn_k.weight' has wrong shape; expected 8192, 8192, got 8192, 1024, 1, 1 llama_load_model_from_file: failed to load model

Do you know what might be causing this? How does one run the 70B parameter model on Mac?

[ScarletEmerald]

Are you still using an old model file? With the current code, it needs GGUF format, so you have to convert the model files again.

[JRZS]

I used the convert script 'convert-llama-ggmlv3-to-gguf.py' to convert the model to gguf, and I get the error:
error loading model: create_tensor: tensor 'blk.0.attn_k.weight' has wrong shape; expected 8192, 8192, got 8192, 1024, 1, 1

[ianscrivener]

"-gqa used to be needed when loading Llama2 70B model, but in the current code with the new GGUF format it is no longer used"
~ @ScarletEmerald