tools: add ClawBench-Lite upstream encrypted-blob builder - #32
Merged
Conversation
Reads test-cases/lite.json, resolves each entry to its task.json, and
produces exactly {task_id, confirmed_task} per task with confirmed_task
byte-identical to test-driver/run.py::build_instruction (verbatim copy
tagged "KEEP IN SYNC"). Fernet-encrypts with seed b"ClawBench_Lite_V1",
matching the decrypt path in browser-use/benchmark/run_eval.py.
Used to regenerate ClawBench_Lite_V1.enc for the upstream PR adding
ClawBench-Lite as a 6th task source to browser-use/benchmark. The
build artifact itself is gitignored; only the reproducible builder
lives in-tree.
PEP 723 inline script metadata so `uv run tools/build_clawbench_lite_enc.py`
installs cryptography on demand without touching test-driver/pyproject.toml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tools/build_clawbench_lite_enc.py— the reproducible builder forClawBench_Lite_V1.enc, the encrypted task blob consumed by browser-use/benchmark. Used to regenerate the blob for the upcoming upstream PR that adds ClawBench-Lite as a 6th task source alongside WebBench / Mind2Web 2 / GAIA / BrowseComp / Custom.test-cases/lite.json, resolves each entry to itstask.json, and emits exactly{task_id, confirmed_task}per task.confirmed_taskis produced by a verbatim copy oftest-driver/run.py::build_instruction(tagged "KEEP IN SYNC with test-driver/run.py::build_instruction (L223-266)"). No prompt editing, no appended sections, no shims — "Lite" is the same 20 tasks, unchanged.b\"ClawBench_Lite_V1\", matching the decrypt path inbrowser-use/benchmark/run_eval.py.uv run tools/build_clawbench_lite_enc.pyinstallscryptographyon demand — does not touchtest-driver/pyproject.toml.Design notes
test-cases/lite.json, so the builder lives alongside it. browser-use/benchmark maintainers receive a pure-data contribution (the.enc) — they never run the adapter.answerfield. ClawBench eval is agentic (seeeval/agentic_eval.md): a generic 9-rule rubric applied by a Claude Code subagent comparing agent trajectory vs human reference trajectory across 5 synchronized layers. Ground truth is the human run, not hand-written PASS criteria. Hardcoding per-taskanswerstrings would be incompatible with this philosophy..gitignorenow excludesclawbench_lite_v1.plaintext.json(from--dry-run) andClawBench_Lite_V1.enc(encrypted output).Test plan
uv run tools/build_clawbench_lite_enc.py --dry-runwrites plaintext with 20 tasks, each having exactly{task_id, confirmed_task}lite.jsonorder (872→002→047→372→369→867→501→486→712→142→469→266→279→091→783→007→809→179→215→403)build_instructionsource text in adapter is byte-identical totest-driver/run.pyL223-266 (verified viadiff)build_instruction(task.json)run (0 mismatches)extra_info— the extras branch is exercisedFernet(sha256(b\"ClawBench_Lite_V1\")).decrypt(encrypt(tasks)) == dry_run_plaintext—) survives round-trip (ensure_ascii=Falseon encrypt path, consistent with dry-run)shared/alex_green_personal_info.jsonor anyextra_info/*file contents — only the manifest + 20task.jsonfilesRelated