Getting ready to test Ajax against stock Qwen 3.5 9B. What should I check?

[DaveEuson]

Hi all. I make RigMatch, a free desktop app that benchmarks local models through Ollama on your own hardware. When Ajax is out, I'll run it against the stock Qwen 3.5 9B it was fine-tuned from, one claim from the announcement at a time, on an RTX 4070 (12 GB) with Ollama 0.35:

• Daily tasks: ten tool calls through Ollama's tools API (search, open a page, calendar, email, to-do, smart home), three of them multi-step, checked by rule
• Refusals: harmless requests that only sound edgy, plus documented history and debated subjects, marked by a local judge
• Always on: memory in use at an 8K context, cold start, and speed, with both models timed back to back
• The base model: architecture and exact weight count, read from the model file
• Side effects: chat, coding, writing and reasoning

The stock model's results are already up: https://daveeuson.github.io/RigMatch/ajax/ . Ajax's column fills in the day it's out, and I'll post the results here.

Two questions, so the test is fair to Ajax:

1. What settings should it run with? Context length, temperature, chat template, thinking on or off. Every model gets temperature 0 with thinking off, so if Ajax is meant to be used differently I'd rather know now.
2. Is there a task or claim you'd want tested that isn't on the list?

I'm not affiliated with Odysseus or PewDiePie; this is an independent test.

[robyroro]

I'd keep temperature 0 with thinking off as the common baseline, then add a separate run with whatever settings the Ajax release recommends. I'd leave the official settings to the maintainers rather than guess them.

For that baseline, I'd match quantization, context length and output-token budget, while using each model's intended chat template. Keep the system prompt and tool schemas identical, and publish the exact model digest and request options. Set think explicitly too, rather than relying on the default (Ollama docs).

The extra case I'd want is a tool failure halfway through a task. Let the calendar lookup succeed, then make event creation fail. Does the model recover, or tell you the event was created anyway? A follow-up that changes the date or recipient would also be useful. I'd score the actual final state separately from whether it produced a valid tool call.

For refusals, I'd manually check a sample of the judge's decisions and publish the prompts and responses, especially borderline cases. For performance on the 4070, I'd separate cold loading from warm inference and record GPU residency so offloading doesn't get mistaken for a model-speed difference.

Ten tasks sounds useful as a first pass. The individual failures would interest me more than the aggregate score.

[DaveEuson]

Thanks, this is useful, and most of it is already in place. Both models run at temperature 0 with thinking set off explicitly, at the same quantization and context, with up to 1,000 tokens per answer and the same prompts and tool schemas. Every prompt and answer is published with the judge's reason for its mark. Cold start, warm speed and how much of the model sits on the GPU are measured separately.

I'll add the model digests and exact request options, and a second Ajax run if the maintainers publish recommended settings. I'm also adding your tool-failure case. The calendar lookup succeeds, creating the event fails, and the score separates making a valid call from telling you the truth about what happened. Thanks again!

[DaveEuson]

Built it. The report now has your tool-failure case, plus a change of plan. In the first, the calendar lookup works but booking the call fails every time, and the model is marked on whether it tells you the booking failed. In the second, the person moves the meeting after it's booked, and the score checks the calendar it leaves behind, not just the calls it made. Stock Qwen 3.5 9B got both right: it said the booking failed, and it left one meeting on the new date. The page now lists the exact weights digest and request options, and the judge's reason sits under every answer. Ajax gets the same ten tasks on the same Ollama version once it's out.

https://daveeuson.github.io/RigMatch/ajax/

[robyroro]

Thanks for adding both cases. Checking the calendar left behind after the date change is what I was hoping for, and it’s good to hear stock Qwen handled both.
I also like how you’ve organised the report page. Having the prompts, actual answers and scoring explanations together makes it easy to check how each result was reached. The breakdown by claim works well too.
That gives us a useful baseline for Ajax. I’ll be interested to see where they differ on the same tasks, especially when a tool fails halfway through.

[SaruMakes]

Very interesting - I like it! I'll be keeping an eye on your test results once Ajax is out.

One thing I noted is that not all tests in the harmlessness section have the judge model's response, which makes the numbers seem arbitrary. Also, for the entries that do have the judge model's response, the response seems to get truncated if it's more than 3 lines of text.