-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Ollama provider #96
Conversation
b3da82a
to
51ad917
Compare
TODO: tests pass when targeted manually but running |
# === TERMINAL A ===
# Checkout and install Ollama as part of tools.
git fresh-checkout ollama
make install-all
# === TERMINAL B
# Ollama server instance which manages all models.
~/.eval-dev-quality/bin/ollama serve &
# === TERMINAL A ===
# Pull some models (check https://ollama.com/library for more but watch out, many are several GBs large).
~/.eval-dev-quality/bin/ollama pull qwen:0.5b
~/.eval-dev-quality/bin/ollama pull tinyllama:latest
# === TERMINAL B ===
# Kill the server, or don't cause our eval sould either start a new one or reuse the existing one.
# === TERMINAL A ===
# Run the eval with only the models we pulled.
eval-dev-quality evaluate --model ollama/qwen:0.5b --model ollama/tinyllama:latest |
I've run the bash script and it gave me the following output: |
89e9eff
to
1311539
Compare
f603a82
to
e256bda
Compare
f44693f
to
cc46ef3
Compare
Early merger is #95.