Skip to content

v0.5.9: the published model's tool calls, as Qwen tokenizes them

Choose a tag to compare

@changkun changkun released this 14 Oct 11:35
· 44 commits to main since this release

Two open items from the v0.5.8 review, measured.

6a: the published model rarely calls the tool. 6a told the published model about the calculator with the <tool_call> markers of its instructions spelled out as characters, where Qwen's own tokenizer makes them single tokens. Written Qwen's way (tools.encode(..., system_markers=True), evaluate_tools.py --system-markers):

  • Calls: it calls in 5.9% of the problems, against 0.5%; 16 of its 260 calls are malformed.
  • Score: 41.5% against 40.2%, a difference within noise.
  • Cost of the tool's description: 5.3 points against no tool (interval 2.7 to 8.1), not 6.6.

The note, its table, figure and film, 9b's exercise and the README now give this run first. The default encoding is unchanged, so the fine-tuned model of 6a is still evaluated as it was trained.

Stage 7: the LoRA model at 320 tokens scores 36.5%, the number in stage 7's table that had no saved file. It is now runs/eval/gsm8k-sft-gsm8k-lora-320.json, with its command in "Run it".

The films of every chapter are attached, 25 of them.