WebGrader

WebGrader constructs executable rewards for web-development reinforcement learning. Given a website requirement and a generated project, it derives the required interaction flows, grounds them against the project and live DOM, executes Playwright programs in Chromium, collects visual, DOM, response, and persistent-state evidence, and produces an evidence-conditioned functional reward.

This repository contains the verifier implementation, the offline self-evolution pipeline, and WebGen-Verifier-100. The promoted SkillGraph used for the paper results is not included. The default command runs the Base Verifier, and skills/examples/ contains a small interface example.

Repository contents

• webgrader/planner: requirement-first Flow Contract construction.
• webgrader/executor: source/DOM grounding, Playwright generation, execution, evidence capture, and optional SkillGraph routing.
• webgrader/judge: evidence-conditioned semantic verdicts.
• webgrader/reward: the scalar reward and VERL aggregation hook.
• evolution: residual attribution, graph mutation, promotion, and freezing.
• benchmark/WebGen-Verifier-100: runnable benchmark projects and annotations.
• training: SFT, GRPO, and reward-aggregation configurations.
• evaluation/statistics: paired bootstrap and significance-test utilities.

Installation

Python 3.10+ and Node.js 20 are recommended.

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
npm ci
npx playwright install chromium
cp .env.example .env

The verifier core uses the Python standard library. Node Playwright is used for browser execution.

Quick verification

bash scripts/run_smoke_test.sh

This compiles the Python modules, runs the offline contracts, and executes the minimal real-browser evidence test when Playwright is installed.

Run WebGrader

Input JSONL records contain page_id, the original instruction or query, and generated_project_dir.

export WEBGEN_SHARED_NODE_MODULES="$PWD/node_modules"
export WEBGRADER_PLANNER_API_KEY="..."
export WEBGRADER_PLANNER_API_BASE="https://provider.example/v1"
export WEBGRADER_PLANNER_MODEL="..."
export WEBGRADER_API_KEY="..."
export WEBGRADER_API_BASE="https://provider.example/v1"
export WEBGRADER_EXECUTOR_MODEL="..."
export DEEPSEEK_API_KEY="sk-REPLACE_WITH_DEEPSEEK_KEY"
export WEBGRADER_GPT_API_KEY="sk-proj-REPLACE_WITH_OPENAI_KEY"

webgrader-run \
  --input-jsonl examples/input.example.jsonl \
  --out-dir outputs/example \
  --root .

On Linux, run inside an isolated container and add --unsafe-no-process-sandbox; macOS uses the local process sandbox when available.

The Planner first derives required flows using only the requirement. The Executor then reads the generated code and live DOM to ground those flows. The public default does not load a promoted SkillGraph. To test the graph interface, pass --skill-graph skills/examples/toy_skillgraph.json.

Reward contract

For each required flow, a requested control that is absent or a target action that the generated website cannot execute is a determinate FAIL. Selector ambiguity, browser infrastructure exceptions, or malformed evaluator output are INCONCLUSIVE. The functional score I is the weighted pass rate over determinate flows. A rollout with no determinate flow receives zero functional credit. The final reward is

R = 0.7 * (5 * I) + 0.3 * A

where A is the appearance score on [0,5]. There is no appearance gate, minimum determinate-weight mask, or rollout mask. An unsuccessful evaluator API job remains incomplete instead of receiving an invented reward.

WebGen-Verifier-100

The repository includes 100 requirements, 100 accepted clean applications, 800 materialized single-fault applications, 478 annotated flows, and the fixed 60/20/20 page split. After qualification, 778 faults and 470 flows are active.

• Train and Eval release runnable projects, flow contracts, fault annotations, and scoring metadata.
• Test releases 180 opaque runnable cases. Test verdicts, flow GT, fault labels, severity, and scoring GT are withheld.
• A clean page and all of its variants remain in the same split. See the dataset card for fields, counts, and evaluation boundaries. The source instructions are sampled from WebGen-Instruct, whose dataset card lists the MIT license.

Training integration

webgrader.reward.verl_reward_worker.compute_score is a VERL-compatible reward aggregation hook. It consumes build/render status, appearance score, and Judge flow results from rollout extra_info. The Test split is not accepted by training or verifier evolution.

Reproducing or extending self-evolution

The code can attribute residuals, induce a graph, generate NAS-inspired mutations, select candidates on Eval, and freeze the selected artifact.

License and citation

Code is released under the MIT License. WebGen-Verifier-100 is distributed under the terms in benchmark/WebGen-Verifier-100/LICENSE; its dataset card also records the upstream attribution. Author and paper metadata are provided in CITATION.cff; the arXiv identifier will be added after posting.