Skip to content

CUA-RSIBench v0.5: Executable Data Research and Sol 6 / Luna 6

Choose a tag to compare

@nanobanana123 nanobanana123 released this 23 Sep 15:54
e536229

A completed, English-language computer-use data-research pilot inspired by RSIBench-Data. Researcher agents author Python factories, generate native Kanboard tasks, collect independently verified GUI demonstrations, and train fresh Qwen3.5-4B LoRA adapters with Tinker. E2B and Harbor execute the student with a separate saved-state verifier.

The four exact researcher IDs are gpt-6-astra, gpt-5.6-sol, gpt-6-sol, and gpt-6-luna, split into two explicitly separate cohorts. The fixed teacher is gpt-5.6-sol. Across 20 research rounds, 13 candidates complete 32-step training and sampling verification, totaling 1,636,954 scheduled training tokens.

In the Sol 6 / Luna 6 extension, the Sol 6-selected student scores 2/12 (16.7%) across two six-task final repetitions. Base scores 0/12; Luna 6 retains that same base and shares its execution evidence. All 24 extension final trials are valid, with no infrastructure errors. These are results for the trained Qwen student, not direct frontier-actor scores.

The original cohort preserves its infrastructure-invalid originals. A separate recovery view reports base 0/12 and Sol 5.6-selected 2/12, alongside the valid Astra originals at 3/12. The retained Astra runs use the Mac controller and the replays use Linux; this is not a platform-matched estimate of training-data effects. Neither cohort supports a general frontier ranking or sustained-RSI claim.

Included: 14-page technical report, Markdown manuscript, separate/combined sanitized evidence, standalone PNG/SVG/PDF figures, an offline interactive explorer, and 68 researcher-authored Python programs with provenance. The report distinguishes authentic issue metadata from synthetic workflow/planning inputs and documents strict JSON-interface failures.

Validation: 145 unit tests and 380 independent evidence/privacy checks passed. Every final PDF page was visually reviewed after layout fixes. Desktop/mobile controls and result tables match the evidence; there are no JavaScript errors or page overflow. Runtime/source/worker, checkpoint, task, archive/member, timing and cleanup bindings were checked before admission.

PDF SHA256: 52896b9a57119c6d357f2480f007567cae1d36bca26ecc8987eadfdbe6fa25b8

The historical v0.4 release remains separate. The measured application profile is native Kanboard with DOM-assisted browser use; original Office, full desktop operation, more research seeds and non-adaptive controls remain future work. Provider dollar totals are unknown.