Skip to content

tools: add ClawBench-Lite upstream encrypted-blob builder - #32

Merged
reacher-z merged 1 commit into
mainfrom
clawbench-lite-upstream-adapter
Apr 12, 2026
Merged

tools: add ClawBench-Lite upstream encrypted-blob builder#32
reacher-z merged 1 commit into
mainfrom
clawbench-lite-upstream-adapter

Conversation

@reacher-z

Copy link
Copy Markdown
Collaborator

Summary

Adds tools/build_clawbench_lite_enc.py — the reproducible builder for ClawBench_Lite_V1.enc, the encrypted task blob consumed by browser-use/benchmark. Used to regenerate the blob for the upcoming upstream PR that adds ClawBench-Lite as a 6th task source alongside WebBench / Mind2Web 2 / GAIA / BrowseComp / Custom.

  • Reads test-cases/lite.json, resolves each entry to its task.json, and emits exactly {task_id, confirmed_task} per task.
  • confirmed_task is produced by a verbatim copy of test-driver/run.py::build_instruction (tagged "KEEP IN SYNC with test-driver/run.py::build_instruction (L223-266)"). No prompt editing, no appended sections, no shims — "Lite" is the same 20 tasks, unchanged.
  • Fernet-encrypts with seed b\"ClawBench_Lite_V1\", matching the decrypt path in browser-use/benchmark/run_eval.py.
  • PEP 723 inline script metadata so uv run tools/build_clawbench_lite_enc.py installs cryptography on demand — does not touch test-driver/pyproject.toml.

Design notes

  • Adapter lives here, not in the fork. Source of truth is test-cases/lite.json, so the builder lives alongside it. browser-use/benchmark maintainers receive a pure-data contribution (the .enc) — they never run the adapter.
  • No answer field. ClawBench eval is agentic (see eval/agentic_eval.md): a generic 9-rule rubric applied by a Claude Code subagent comparing agent trajectory vs human reference trajectory across 5 synchronized layers. Ground truth is the human run, not hand-written PASS criteria. Hardcoding per-task answer strings would be incompatible with this philosophy.
  • Build artifact gitignored. .gitignore now excludes clawbench_lite_v1.plaintext.json (from --dry-run) and ClawBench_Lite_V1.enc (encrypted output).

Test plan

  • uv run tools/build_clawbench_lite_enc.py --dry-run writes plaintext with 20 tasks, each having exactly {task_id, confirmed_task}
  • Task order matches lite.json order (872→002→047→372→369→867→501→486→712→142→469→266→279→091→783→007→809→179→215→403)
  • build_instruction source text in adapter is byte-identical to test-driver/run.py L223-266 (verified via diff)
  • For all 20 tasks, adapter output matches an independent build_instruction(task.json) run (0 mismatches)
  • 12/20 tasks have extra_info — the extras branch is exercised
  • Encrypt round-trip: Fernet(sha256(b\"ClawBench_Lite_V1\")).decrypt(encrypt(tasks)) == dry_run_plaintext
  • Em-dash () survives round-trip (ensure_ascii=False on encrypt path, consistent with dry-run)
  • Adapter does NOT read shared/alex_green_personal_info.json or any extra_info/* file contents — only the manifest + 20 task.json files

Related

Reads test-cases/lite.json, resolves each entry to its task.json, and
produces exactly {task_id, confirmed_task} per task with confirmed_task
byte-identical to test-driver/run.py::build_instruction (verbatim copy
tagged "KEEP IN SYNC"). Fernet-encrypts with seed b"ClawBench_Lite_V1",
matching the decrypt path in browser-use/benchmark/run_eval.py.

Used to regenerate ClawBench_Lite_V1.enc for the upstream PR adding
ClawBench-Lite as a 6th task source to browser-use/benchmark. The
build artifact itself is gitignored; only the reproducible builder
lives in-tree.

PEP 723 inline script metadata so `uv run tools/build_clawbench_lite_enc.py`
installs cryptography on demand without touching test-driver/pyproject.toml.
@reacher-z
reacher-z merged commit acb94de into main Apr 12, 2026
@Perry2004
Perry2004 deleted the clawbench-lite-upstream-adapter branch April 23, 2026 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant