[Feature] Add llama 3 model

[qiweiii]

Feature Request

Are we going to support llama 3 model: https://github.com/meta-llama/llama3

[woheller69]

for me it works, but there is an issue: after the first answer the end of the answer does not seem to be detected, CPU stays at 100%...
Then a second question will not be answered. If you stop the model answer before the end it works.

[davidsilvasmith]

Yes very much hoping for Llama3 in GPT4all!

[woheller69]

This fixes the issue for me with GGUFs:
https://www.reddit.com/r/LocalLLaMA/comments/1c7dkxh/tutorial_how_to_make_llama3instruct_ggufs_less/

Problem: Llama-3 uses 2 different stop tokens, but llama.cpp only has support for one. The instruct models seem to always generate a <|eot_id|> but the GGUF uses <|end_of_text|>.

Solution: Edit the GGUF file so it uses the correct stop token.

./gguf-py/scripts/gguf-set-metadata.py /path/to/llama-3.gguf tokenizer.ggml.eos_token_id 128009

[davidsilvasmith]

Thank you, very helpful! I'm on an M1 Macbook Air with 16GB Ram.

I downloaded this model https://huggingface.co/lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF/blob/main/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf and put it in ~/Library/Application\ Support/nomic.ai/GPT4All/ directory and it's working great at about 4.1 - 4.4 tokens per second with my RAM full and about 7.5GB swap.

[davidsilvasmith]

Maybe I spoke too soon, in another test it kept talking and didn't stop until I told it to. So maybe I need to mess with the correct stop token too.

[woheller69]

No, I think then you need to change the max number of tokens you want, or manually press stop

[dontcryme]

fixed instruct model link : https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF
I tested above model and 100% working.