v1.1.0: DPO on math problems

A new side topic, 8b: DPO on math problems. Stage 8's DPO runs on the 0.5B model, and each pair is two of its own replies to a GSM8K training problem: a right one, preferred, and a wrong one, as judged by stage 9's verifier. The model is stage 6's fine-tuned one, which self-training (9a) and GRPO (stage 9) also start from.

• Pairs. 3,840 replies to 480 problems, 32.9% of them right, gave 551 pairs from the 331 problems that had both right and wrong replies.
• No setting gains. Five DPO runs were scored on the 1,319 test problems, against 36.6% at the start:
  • learning rate 2e-6: 35.7%
  • learning rate 1e-5: 35.2%
  • learning rate 5e-5, one pass: 34.6%
  • learning rate 5e-5, two passes: 29.2%
  • learning rate 5e-5 with the right reply's own loss added: 33.8%
• Both replies can fall. At learning rate 5e-5, after two passes, the wrong replies were 91 nats less likely than at the start, and even the right ones were 21.5 nats less likely. Adding the right reply's own loss keeps the right replies up.
• Against GRPO. GRPO sampled as many replies, but drew them as it trained, and reached 42.5%.

New code: minllm/dpo.py, make_preferences.py --task gsm8k, and analyze_dpo.py. dpo.py now carries the 0.5B model.

The films of every chapter are attached, 26 of them.