Application of the autoresearch loop to verifiable CoT generation (programmatic solvers + twin-blind checks)

[habanwer]

Hello!

I applied the core autoresearch pattern (agent iteratively improves one file under a fixed evaluation + keep/discard rule) to a different domain: generating short, fully verifiable Chain-of-Thought traces for the NVIDIA Nemotron Reasoning Challenge (Alice-in-Wonderland puzzles).

Instead of the agent generating free-form CoT, it iteratively improves pure-Python solvers. Every candidate is evaluated with a twin-blind check (the solver is prohibited from seeing or conditioning on the ground-truth answer). Only solutions that pass both correctness and the blind check are kept. The result is short, re-executable reasoning traces rather than model-generated text.

The approach was cited twice in the official NVIDIA blog for emphasizing verified traces over unfiltered scale:
https://developer.nvidia.com/blog/lessons-from-the-leaderboard-what-5000-kagglers-taught-us-about-improving-ai-reasoning/

Code: https://github.com/The-Adimension/Agent-ProSAT
Dataset: https://huggingface.co/datasets/The-Adimension/prosat-nemotron-alice-in-wonderland-traces

Has anyone else experimented with using the same keep/discard loop for synthesizing training data (especially reasoning traces) rather than for model training itself?

I’m particularly interested in better evaluation metrics for programmatic solvers — something more informative than simple coverage, ideally including per-puzzle-type regression and trace quality.

Best,
Shehab Anwer