You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Challenge: what's the smallest local model that passes Odysseus's own tool-use eval, per VRAM tier?
#7288
🪜 Challenge: what's the smallest local model that passes Odysseus's own tool-use eval, per VRAM tier?
The ROADMAP asks for "tool selection for constrained context windows" and for model discovery that ranks by "expected reliability". Today HW Fit's quality score comes from parameter count and name heuristics (services/hwfit/fit.py_quality_score), and people keep asking which local models actually drive tools well (#2670, #5492, #2724). There's no shared, reproducible table.
The repo already has the harness for it: scripts/eval_odysseus_tool_use.py sends 10 tool cases (notes, calendar, tasks, documents, memory, …) and, with --include-no-tool, 5 no-tool cases through the real chat route, scores each from the SSE events, and prints a SUMMARY line with counts only.
First, a question for maintainers 🙋
This script looks like internal tooling. Is community use welcome, and would a small PR making it runnable outside the dev setup be accepted? What blocks a newcomer today:
_cookie() looks up a session for a hard-coded username ("pewds"), so any other admin name fails with "No valid … session cookie found".
--base-url defaults to port 7011, not 7000.
Fixture setup/cleanup opens the database directly, so it must run with the same data dir as the instance under test.
A --username flag plus a short usage note at the top would make it one command.
Scoreboard
execution_success_evaluable and response_quality_success_evaluable from the SUMMARY line, per model and quant. Bonus column: the input_tokens the casual_hi case reports, i.e. what a bare "hi" costs before the user asks for anything (prompt-size work itself belongs in #2750 / #4778; this column just feeds them data).
| VRAM tier (8 / 12 / 16 / 24 GB / CPU) | backend + build | model repo | quant | ctx | execution ok (A, B) | response quality ok (A, B) | input_tokens on casual_hi | Odysseus commit |
Baseline: none yet. The first row in each tier sets it. The fun results: the smallest model that passes everything in a tier, and the tier where nothing does.
First evening: start a throwaway instance with its own ODYSSEUS_DATA_DIR, log in once, serve a model you already have, then from the repo root (same ODYSSEUS_DATA_DIR exported):
Run it twice and post the two SUMMARY lines as one row.
⚠️ Please
Never point it at your daily instance: it creates and deletes ODY-EVAL-TOOL-* notes, tasks, documents and calendar entries, and one case asks for your latest email.
Post counts only — the output JSON contains model responses.
This thread proposes no change to the agent runtime, prompts or tool selection (Harness V2 freeze). It's data for those decisions.
Drafted with the help of an AI agent; script details checked against dev@0096533c.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
🪜 Challenge: what's the smallest local model that passes Odysseus's own tool-use eval, per VRAM tier?
The ROADMAP asks for "tool selection for constrained context windows" and for model discovery that ranks by "expected reliability". Today HW Fit's quality score comes from parameter count and name heuristics (
services/hwfit/fit.py_quality_score), and people keep asking which local models actually drive tools well (#2670, #5492, #2724). There's no shared, reproducible table.The repo already has the harness for it:
scripts/eval_odysseus_tool_use.pysends 10 tool cases (notes, calendar, tasks, documents, memory, …) and, with--include-no-tool, 5 no-tool cases through the real chat route, scores each from the SSE events, and prints aSUMMARYline with counts only.First, a question for maintainers 🙋
This script looks like internal tooling. Is community use welcome, and would a small PR making it runnable outside the dev setup be accepted? What blocks a newcomer today:
_cookie()looks up a session for a hard-coded username ("pewds"), so any other admin name fails with "No valid … session cookie found".--base-urldefaults to port 7011, not 7000.A
--usernameflag plus a short usage note at the top would make it one command.Scoreboard
execution_success_evaluableandresponse_quality_success_evaluablefrom theSUMMARYline, per model and quant. Bonus column: theinput_tokensthecasual_hicase reports, i.e. what a bare "hi" costs before the user asks for anything (prompt-size work itself belongs in #2750 / #4778; this column just feeds them data).Baseline: none yet. The first row in each tier sets it. The fun results: the smallest model that passes everything in a tier, and the tier where nothing does.
First evening: start a throwaway instance with its own
ODYSSEUS_DATA_DIR, log in once, serve a model you already have, then from the repo root (sameODYSSEUS_DATA_DIRexported):Run it twice and post the two
SUMMARYlines as one row.ODY-EVAL-TOOL-*notes, tasks, documents and calendar entries, and one case asks for your latest email.Drafted with the help of an AI agent; script details checked against
dev@0096533c.All reactions