Issues with Llama 3

[dezoito]

If you are having issues with Llama3 or any of its versions, here's the most straightforward to get it running:

1. Use this as your model file for Llama3 (make sure to use the name for the version you've pulled):

FROM llama3:8b
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|reserved_special_token|>"
PARAMETER stop "assistant"
PARAMETER stop ".assistant"
PARAMETER stop "!assistant"
PARAMETER stop "?assistant"

2. On the settings model in Ollama Grid Search, make sure to completely remove the stop parameter from the "default options" textarea:

(The above is what I am currently using)

But why is this necessary?

As far as I can understand, Llama3 requires these specific stop parameters, but there were some issues or misspellings with the models provided from Ollama's library (maybe an upstream issue).

If those are not setup correctly (and you don't specify a num_predict option) , the model will not stop generating tokens (and thus run indefinitely or timeout).

If you pass the API anything for the stop parameter, it will try to use that, instead of what's in the modelfile, and run into the same previous issues.

That was not a requirement for any of the previous models I've tested (and that's a long list), but currently this is the best way to deal with it.

@zedmango