Web Skill Factory: evolving reusable, verified, code-native skills for web agents - #54
Closed
Demi Wang (DEM1TASSE) wants to merge 190 commits into
Closed
Web Skill Factory: evolving reusable, verified, code-native skills for web agents#54Demi Wang (DEM1TASSE) wants to merge 190 commits into
Demi Wang (DEM1TASSE) wants to merge 190 commits into
Conversation
…tool
A built-in submodule turning solved tasks into reusable, executable code skills:
- skills/{library,retrieve,decide,gate,update,llm}: store / retrieve (relevance) /
decide (use·adapt·skip utility) / admission gate (gold|self_verify|none) /
evolve (incremental growth on existing library) — backend-agnostic via configure_llm
over webwright's own Model abstraction (no hardcoded gateway/key/path)
- tools/skill_use.py: solve-time tool (agent invokes like self_reflection/image_qa) ->
retrieve+decide -> JSON recommendation (use/adapt/skip + source path)
- python -m webwright.skills.update --manifest batch.json --library ./lib : batch growth
- tests/skills: 5 unit tests pass against the migrated module (logic == original)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…skill_use CLI - skills/prompt.with_skill_hint: prepend skill-library usage hint to task prompt (non-invasive; webwright merges system_template by replacement, so prompt-level is the clean way) - config/skill_mode.yaml: optional overlay doc + step budget for skill-reuse runs - llm._model(): bare CLI (python -m webwright.tools.skill_use) builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env -> same backend as agent, no hardcoded gateway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: what the module is, the two plug points (skill_use tool + update CLI), components table, gate semantics, backend config, results summary - llm._model(): bare CLI builds model from SKILL_MODEL_NAME/ENDPOINT (or OPENAI_*) env Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
- README: Skill Library section (what it is, reuse via skill_use tool, grow via update CLI, end-to-end validation summary) - tests/skills: 5 unit tests for library/gate/update/evolve/retrieve+decide Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
Remove _grow / update() / _UPDATERS dispatch — evolve() is the single entry now; drop the test_update test that exercised the removed grow path. Keep retrieve/llm fallbacks (useful). Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…val) Three bugs hit when update.refine emits a large skill on a slow gateway: - llm() ignored max_tokens -> model default ~4000 truncated the refined skill mid-code - llm() had no timeout override -> model default 120s ReadTimeout'd on the ~16k-token refine (now request_timeout_seconds defaults 600, env SKILL_MODEL_TIMEOUT) - _extract_code returned raw text (with ```python fence) when the closing fence was missing (truncated) -> skill failed to compile; now strips the opening fence anyway Co-Authored-By: Demi Wang <86202027+DEM1TASSE@users.noreply.github.com>
…lve-time reuse, direct skill run) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
@microsoft-github-policy-service agree company="Microsoft" |
…te+manifest -> update -> reuse); fix output_schema examples to gate's {type} form
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bArena numbers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- traces_from_manifest: 'admit' is now REQUIRED per run — a missing gate verdict raises instead of silently defaulting to admitted (was the main pollution risk) - _slug: templates longer than 48 chars get a short content-hash suffix so two templates sharing a long prefix can no longer overwrite each other's skill - skill_use.recommend: the decision's skill_id must be one of the RETRIEVED candidates; anything else (LLM hallucination, even an existing library id) downgrades to skip - tests for all three Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… is truthy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tive paths, missing answer file) Independent cleanroom reproduction (fresh clone + venv, README-only, public GitHub tasks) surfaced usability failures; mechanism itself reproduced end-to-end in 25 min. - with_skill_hint resolves the library path to ABSOLUTE (F2): the hint's command runs in the agent's workspace, where a relative ./library silently resolved to a nonexistent dir -> empty library -> every lookup skipped, no error, answer still right - skill_use.recommend: a missing/empty library now answers skip with an explicit 'warning: library empty at <abspath>' BEFORE Library() can mkdir the bogus path (F3) - README: step 1 now tells the agent to write agent_response.json (stock webwright does not produce it; the gate/manifest flow assumed it) with a copyable ANSWER_SPEC (F1); absolute-path + --library-beats-env notes (F2/F4); custom endpoint tip (F5) - skills/__init__ no longer eagerly imports update -> no more runpy RuntimeWarning on 'python -m webwright.skills.update' (F6); import evolve/Trace from the submodule - tests: hint abspath, empty/missing-library warning (incl. no-mkdir side effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, filled-in inputs - example_library/: the commit-counting skill verbatim as evolve wrote it (runnable standalone via taskspec, no LLM in the loop; functionally verified against a local repo) - README: what a skill looks like (catalog card + the distilled git-log algorithm), measured held-out numbers (33->10 steps; wrong->correct rescues; honest note that reuse costs more than it saves on cheap tasks), three try-it paths - honest coverage-boundary demo: an unseen period shape raises cleanly; on the real held-out run the agent read the source and adapted around it - tasks/batch/taskspec example JSONs matching the how-to-use steps - links from the module README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, safe growth, measured results) + data-flow/interfaces diagram
…no manifest) python -m webwright.skills learn <runs_dir> --library ./library - auto: reads task.json/agent_response.json per run, gates (gold via --golds, else self_verify), groups tasks into templates + extracts params with one LLM call per chunk (default 25; existing templates passed in so chunks refine instead of duplicating), infers output_schema from the answer shape, site from start_url - idempotent: processed runs remembered in library/.learned.json; --dry-run plan mode - README: Quickstart (two commands) + use cases up top; old walkthrough demoted to 'Manual mode'; examples/solve_with_library.sh wrapper (hint + answer instruction) - unit tests for the LLM-free plumbing
…ks, leaks) External-user test of the friendly path surfaced that a trivially-easy config mistake (gateway key + unset OPENAI_ENDPOINT) silently disabled ALL reuse. Fixes: - skill_use: a hard error still degrades to skip (never block solving) but now says LOUDLY it is a LOOKUP FAILURE, not a no-match — error field in the JSON, hint about OPENAI_ENDPOINT/SKILL_MODEL_ENDPOINT, and a stderr line (F1+F2) - README Quickstart: gateway users must export OPENAI_ENDPOINT/OPENAI_MODEL for BOTH steps, stated where step-1 users actually look (F2) - learn: grouping-LLM failure now exits with a one-line actionable message instead of a 40-line traceback (F3); skipped-for-missing-answer runs get a visible summary with the correct pointer (the old message named a command that does not exist) (F4) - learn: strips the answer-output instruction from task text so it cannot leak into templates/skill_ids (F7) - solve_with_library.sh: usage check instead of passing empty args into the CLI (F6)
…ers get them too)
…ssion tests - README: "Only verified solves get in" -> "Validation-gated, exactly as strong as the gate you give it" — states plainly that the default self_verify checks shape only and that the WebArena numbers used the gold gate; learn prints the same warning at run time when no --golds is given - examples/learned_library/: a skill produced by "skills learn" from 3 real GitHub solves — n_solves=3, owner/repo lifted to parameters, two extraction strategies as fallbacks; verified standalone on an unseen repo (numpy/numpy -> v2.5.1, no model); test_learned_example.py locks n_solves>=3 + lifted params + no leak - regression tests for the interface-test findings: F1 (skill_use surfaces hard errors as ERROR, not quiet skip), F3 (learn exits with an actionable message)
There was a problem hiding this comment.
Pull request overview
This PR introduces a new webwright.skills subsystem that turns previously solved tasks into reusable, executable “skills”, enabling solve-time reuse (via a CLI tool) and offline library growth (via learn/update pipelines) while keeping the main agent loop unchanged.
Changes:
- Adds a disk-backed skill library (
Skill/Library) plus retrieve/decide/gate/evolve/learn modules to store, select, admit, and incrementally refine skills. - Adds
webwright.tools.skill_useas a solve-time CLI that recommendsuse|adapt|skipand provides the source path for reuse. - Adds docs/config/examples and new tests to validate deterministic plumbing and example artifacts.
Reviewed changes
Copilot reviewed 29 out of 31 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/skills/test_retrieve_decide.py | Adds deterministic tests for retrieve/decide + skill_use/prompt behavior (currently not pytest-discoverable). |
| tests/skills/test_library.py | Adds tests for disk persistence of Library (currently not pytest-discoverable). |
| tests/skills/test_learned_example.py | Adds a check that the checked-in learned example is aggregated/parameterized (currently not pytest-discoverable). |
| tests/skills/test_learn.py | Adds tests for learn plumbing + regression handling (currently not pytest-discoverable). |
| tests/skills/test_gate.py | Adds tests for gate admission logic (currently not pytest-discoverable). |
| tests/skills/test_evolve.py | Adds tests for evolve behavior and slug collision avoidance (currently not pytest-discoverable). |
| src/webwright/tools/skill_use.py | Introduces solve-time library recommendation tool with guardrails against missing/empty libraries and hallucinated skill IDs. |
| src/webwright/skills/update.py | Implements incremental library evolution and refinement prompt construction + manifest ingestion. |
| src/webwright/skills/retrieve.py | Implements LLM-based retrieval plus a simple deterministic keyword-overlap fallback. |
| src/webwright/skills/decide.py | Implements LLM-based use/adapt/skip decision over retrieved candidates. |
| src/webwright/skills/gate.py | Implements admission gate (gold/self_verify/none/auto) to prevent wrong solves from entering the library. |
| src/webwright/skills/learn.py | Adds “friendly” pipeline to learn skills from run folders with gating, chunked grouping, and an idempotent ledger. |
| src/webwright/skills/library.py | Adds on-disk skill storage (<id>/skill.py + meta.json) and simple list/get/add APIs. |
| src/webwright/skills/llm.py | Adds backend-agnostic LLM helper using Webwright’s Model abstraction. |
| src/webwright/skills/prompt.py | Adds with_skill_hint() helper that prepends a bash command hint to consult the skill library. |
| src/webwright/skills/init.py | Exposes the public webwright.skills API surface for consumers. |
| src/webwright/skills/main.py | Adds `python -m webwright.skills <learn |
| src/webwright/skills/README.md | Adds comprehensive module documentation, usage patterns, and rationale. |
| src/webwright/skills/pipeline_diagram.svg | Adds diagram documenting data flow and interfaces for the skills pipeline. |
| src/webwright/config/skill_mode.yaml | Adds optional config overlay to increase step budget for skill reuse runs. |
| src/webwright/skills/examples/README.md | Adds examples overview and how-to for running skills/tools and batch pipeline. |
| src/webwright/skills/examples/solve_with_library.sh | Adds helper script to prepend hint + answer spec and run Webwright with a library. |
| src/webwright/skills/examples/taskspec.example.json | Adds example taskspec input for running a skill standalone. |
| src/webwright/skills/examples/tasks.example.json | Adds example batch task list input with params/golds. |
| src/webwright/skills/examples/batch.example.json | Adds example manifest for update (admit/params/schema/etc). |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/skill.py | Adds a runnable example skill produced by the pipeline. |
| src/webwright/skills/examples/example_library/how_many_commits_did_user_make_period_in_the_cur/meta.json | Adds metadata for the example skill. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/skill.py | Adds a checked-in “learned” skill example aggregated from multiple solves. |
| src/webwright/skills/examples/learned_library/what_is_the_latest_release_version_of_ow_c29dab8/meta.json | Adds metadata for the learned skill example. |
| README.md | Adds top-level README section linking to the new skill-library capability and docs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| f"answer={json.dumps(tr.answer, ensure_ascii=False)[:120]})\n```python\n{tr.code}\n```" | ||
| ) | ||
| sys_prompt = _REFINE_SYS + (_REFINE_INCREMENTAL if existing else "") | ||
| code = _extract_code(llm(sys_prompt, "\n\n".join(blocks), max_tokens=16000, timeout=400)) |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — removed the kwarg (it was indeed swallowed by llm()'s **_). Timeout is governed by the model's request_timeout_seconds (600s default in _model()).
Comment on lines
+175
to
+179
| if answer is None and (d / "agent_response.json").exists(): | ||
| try: | ||
| answer = json.load(open(d / "agent_response.json")).get("retrieved_data") | ||
| except Exception: | ||
| pass |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+198
to
+199
| manifest = json.load(open(a.manifest, encoding="utf-8")) | ||
| traces = traces_from_manifest(manifest) |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — switched to Path.read_text(encoding="utf-8").
Comment on lines
+59
to
+64
| def llm_json(system: str, user: str, **kw: Any) -> dict: | ||
| """Call + parse the first {...} JSON object out of the reply.""" | ||
| txt = llm(system, user, **kw) | ||
| match = re.search(r"\{.*\}", txt, re.S) | ||
| if not match: | ||
| return {} |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — llm_json now walks the reply with json.JSONDecoder().raw_decode and returns the first valid JSON object, skipping non-JSON brace snippets; matches the docstring.
Comment on lines
+48
to
+56
| meta = json.loads((d / "meta.json").read_text()) | ||
| code = (d / "skill.py").read_text() if (d / "skill.py").exists() else "" | ||
| return Skill(skill_id=skill_id, code=code, meta=meta) | ||
|
|
||
| def add(self, skill: Skill) -> None: | ||
| d = self._dir(skill.skill_id) | ||
| d.mkdir(parents=True, exist_ok=True) | ||
| (d / "skill.py").write_text(skill.code) | ||
| (d / "meta.json").write_text(json.dumps(skill.meta, ensure_ascii=False, indent=2)) |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — explicit UTF-8 on all skill/meta reads and writes in library.py (and the same for learn.py's ledger/run files).
Comment on lines
+34
to
+35
| if __name__ == "__main__": | ||
| run() |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+86
to
+87
| if __name__ == "__main__": | ||
| run() |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+26
to
+27
| if __name__ == "__main__": | ||
| run() |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+69
to
+70
| if __name__ == "__main__": | ||
| run() |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all).
Comment on lines
+53
to
+55
| if __name__ == "__main__": | ||
| run() | ||
| run_regressions() |
Contributor
Author
There was a problem hiding this comment.
Fixed in 6513d87 — added a pytest entry point (test_all) that runs both run() and run_regressions().
The README table (held-out 70% vs 55%, 14.7 vs 17.1 steps; train 26/30 vs
23/30) previously existed only as prose. Now:
- evals/webarena/results/: sanitized per-task records of the exact run behind
the table — task id, answer, gold score, steps, skill verdict; one command
("reproduce.py table --results results") re-derives the table, no setup
- evals/webarena/reproduce.py: self-contained driver that re-runs the whole
experiment (train -> gold-gated update -> held-out with/base -> table)
against your own WebArena deployment via microsoft/webarena-verified;
resumable, parallelizable per template
- run_all.sh + model.eval.yaml (agent-model overrides; gateway pointer)
- tests/skills/test_eval_snapshot.py: CI-locks the records to the published
numbers and enforces snapshot sanitization (no local paths/hosts/keys)
- skills README + examples README now link the records instead of asking for
trust; CI also triggers on evals/webarena/**
- prompt.py: shell-quote task and library in the skill_use hint (shlex.quote) —
$VAR / $(...) / backticks expanded in bash even inside the old double quotes;
regression test added
- llm.py: llm_json now scans for the FIRST valid JSON object (raw_decode) as
documented, instead of a greedy first-{ to last-} regex that could span
unrelated braces
- update.py: drop the misleading llm(..., timeout=400) kwarg (silently
swallowed; the model's request_timeout_seconds already governs); read JSON
files via read_text instead of unclosed open()
- library.py / learn.py: explicit UTF-8 on every skill/meta/ledger read+write
(locale-independent on Windows)
- tests: pytest-discoverable test_all() entry points in all 7 files (CI keeps
running them as scripts too)
Contributor
There was a problem hiding this comment.
what is the purpose of this file? Demi Wang (@DEM1TASSE)
Contributor
Author
There was a problem hiding this comment.
It's the guided tour of the examples/ directory (the module README links here for every "see examples"): two real, checked-in skill libraries — learned_library/ is the Quickstart loop's actual output (3 GitHub solves -> learn -> owner/repo lifted to parameters, runs standalone on unseen repos with no model), example_library/ is verbatim update.evolve output from the WebArena eval — plus the solve wrapper and filled-in copies of every input file the manual pipeline asks you to write. 8ea59be makes this explicit in the file's opening paragraph and adds the learned_library provenance section.
Contributor
There was a problem hiding this comment.
this file looks redundant, can we remove?
Contributor
Author
There was a problem hiding this comment.
Agreed — removed in 8ea59be. Nothing referenced it: the skill hint is prompt-level (with_skill_hint), and no documented path needed the step_limit bump.
| # 2. turn everything you've solved into skills — no manifest, no fields to learn | ||
| python -m webwright.skills learn outputs/ --library ./library | ||
| ``` | ||
|
|
Contributor
There was a problem hiding this comment.
Demi Wang (@DEM1TASSE) It is better to add the complete example in the quick start session.
It needs to additionally include how to use the skill library.
Contributor
Author
There was a problem hiding this comment.
Done in 8ea59be — the Quickstart is now the complete loop on a copy-pasteable example (public GitHub): solve 3 instances -> learn -> an unseen instance reuses the skill (with the expected skill_decision.json shown), plus how to use the library without the agent (querying skill_use directly, and running the learned skill standalone with no model — verified pandas-dev/pandas -> v3.0.4). The same loop's output is checked in at examples/learned_library/.
…larify examples/ - README Quickstart is now the complete loop on a runnable example (public GitHub): solve 3 instances -> learn -> an UNSEEN instance reuses the skill, plus using the library without the agent (skill_use query, running the learned skill standalone, verified: pandas-dev/pandas -> v3.0.4 with a params-only taskspec) - remove config/skill_mode.yaml: nothing referenced it (the skill hint is prompt-level via with_skill_hint; the step_limit bump was never needed by any documented path) - examples/README: state the directory's purpose up front, and document learned_library's provenance (the Quickstart loop's checked-in result) with the unseen-repo runs and the CI test that locks it
…ed wording Replaced the comparison with the reviewed version verbatim — it reads cleanly and every cell checks out against source. One fix only: the "how it's created / Ours" cell had a "[by llm/pipeline?]" placeholder; resolved it to what the code does — an LLM distills the runs into one program (update.py _refine). Moved it out of its own top-level section and into a collapsed <details> under How it works, right before "The Quick Start below demonstrates the complete workflow," so the section stays scannable and the comparison is there for anyone who opens it. Dropped the source-citation footnote in favor of a plain disclaimer: this is a friendly comparison based on our understanding; if we've mischaracterized a project, open an issue or PR and we'll fix it.
The "what a skill is" row said "the model reads" for SKILL.md and SkillOpt but "an agent reads" for OpenSpace — same meaning, inconsistent wording, no reason for it. Unified to "the model reads" (kept OpenSpace's helper-files nuance), which is also the axis this row is about: all three are documents that need the model to run, versus ours, a program that runs with no model.
"an LLM distills them into one program" was too narrow. Creating a skill is a pipeline: an input gate filters wrong solves, an LLM groups the runs and writes the program, then a deterministic model-free replay-verify keeps it only if it reproduces the recorded answers (learn.py -> update._refine -> _replay, with a draws x rounds retry). The LLM writes the code, but the pipeline — and especially the verify step — is the point. Reworded to say so.
…iewed wording Consolidated to four rows and applied the reviewed fixes: - "how it's created" and "what grows from your runs" merged into one honest row, "how the library evolves from runs" — ours also grows the library (add a skill for a new template, leave it unchanged on a plain reuse, refine it on an adapted run), not just "widens one skill." Same for the others, stated plainly: OpenSpace fixes/derives/captures documents; OpenCLI authors maintain commands. - "how it handles variation" -> "how one skill covers different inputs," asking the concrete question: does the skill take real inputs, and who set them. Ours: aligned verified runs, differences become explicit parameters. Others: prose the model improvises from, or hand-declared arguments. - "no model needed" -> "no model required (an agent can still use it)" — the skill runs standalone, but can also be used by an agent; the distinction from the document rows is that those always require a model. - Dropped the jargon "fixture" for "a saved expected result." - "the author" -> "a person or agent" for OpenCLI authoring/maintenance — confirmed in source: opencli-adapter-author is explicitly an agent workflow and autofix auto-repairs adapters. Ours cells unchanged in substance; the differentiators still stand once the neighbors are described accurately.
…read - "no model required" is scoped to execution: an executable program, no model to run it — but an agent (or you) still picks the skill and supplies its inputs, so no one asks who selects and parameterizes. - published SKILL.md's variation cell now reads naturally and keeps the runtime fact: the model interprets the author's written guidance for each input. - OpenCLI fixtures are output constraints (format / non-empty / row count), not an exact saved value — reworded from "a saved expected result." - OpenSpace verification stated as a plain trust lifecycle: not a correctness check; a new skill starts untrusted, earns trust by working on real tasks, and loses it after a failure it caused. - ours refine "may become parameters" (not always), and can also repair or harden the skill — the _REFINE_SYS prompt explicitly asks for robust extraction.
Added clickable links on the column headers — published SKILL.md -> anthropics/skills, SkillOpt -> microsoft/SkillOpt, OpenCLI -> jackwener/OpenCLI, OpenSpace -> HKUDS/OpenSpace. All four verified reachable. Ours is this repo, so no link.
…trust step "checked before use, starts out untrusted" read as a contradiction (why still untrusted after a check?). They're two different bars. Now the cell says what admission actually checks — the skill is well-formed and doesn't regress the library (behavior_eval's contract/routing/replay gates), not correctness — and then that it stays untrusted until real successful use promotes it to trusted, with a failure demoting it. Two bars, clearly separated.
…ault
Route/run/recommend as one symmetric entry point:
- route.py: judge -> act -> fallback in one call. verdict {run, adapt, skip};
run executes a matching executable skill directly (no agent), falling back to
the agent as an adapt on crash/timeout/empty/wrong-shape. adapt hands the task
to the agent with the skill as a prior; skip solves from scratch.
- recommend() (skill_use) now emits run/adapt/skip out-of-loop; promote() maps the
in-loop use verdict to run (executable + all slots fill) or adapt.
- execute.py/fill.py/entry_shim.py: direct skill exec via taskspec, slot filling,
and the auto-prepended CLI shim so generated skills take --flags AND taskspec.json.
Defaults: on_fail now reference (never empty-handed — a failed replay lands as a
readable grade=reference prior the agent can adapt) across build/learn/update
functions + CLIs and the init spec skeleton; reject stays as the stricter opt-in.
init auto-proposes real, varied instance values for review.
agent_cfg prepends base.yaml when -c omits it, so -c model.yaml alone works.
README + docs/skill_factory/reference.md rewritten for route/run and the new
default; quickstart.sh modes are run/route/solve. Full skill_factory suite green.
…ault - Quick Start "What that just saved": replace the single-run 50/11/10 table with the paired n=5 with-vs-without numbers (mean/std/worst), keeping the standalone floor (~10 steps / ~40s / 0 model) as its own labelled column; remove the now-duplicate n=5 block from Results. - examples/README + quickstart.sh: stale mode names ask/demo -> run/route/solve; drop the orphaned 50/11 single-run figures, point to the module README.
…-loop framing - Remove examples/quickstart.sh; §1/§2 now use direct commands: run the checked-in skill.py with --flags (no model), and `route` for the agent path (prints the decision, then runs the skill directly or launches the agent). - How it works: correct the stale in-loop description — reuse is resolved OUT of the agent loop (result injected into the prompt, or run directly by route), not an in-loop skill_use Bash tool the agent queries. Explain `recommend` as the pure decision route is built on (not a command, not an in-loop tool). - Purge every quickstart.sh / MODEL_CFG / QUICKSTART_WORKDIR reference across README, examples/README, trajectories/README, reference.md. - Unify the route example start-url with the checked-in spec (google.com/flights).
…nual/reference - README §3: replace the Amazon init example with the flights one (strict/stable, note a drifting answer would pick verify: shape); matches the running example. - manual.md §4: reuse is resolved out of the agent loop — with_skill_hint injects the chosen skill into the prompt; the agent never queries the library itself. - reference.md: document `route` (the out-of-loop entry) alongside skill_use; reframe skill_use as the `recommend` decision; add route/execute/fill/entry_shim/ recommend to the components table; fix the SKILL_LIBRARY_ROOT / prompt.py wording. - gitignore the root-level generated ./library.
… the agent - Move "What that just saved" out of §1 (Run a learned skill) into §2 (Bring the agent in), where the with/without comparison belongs — it's the agent-adapt path. - §2 now opens with WHY the agent is needed: the shortest-duration task doesn't fully match the earliest-departure skill, so route decides adapt. - Label the standalone column "earliest nonstop departure" and add the comparability note: different question, nearly identical work (one filter over the same results table), so it's a fair floor for the same machinery.
…agrams - root README: the Reuse bullet described the old in-loop skill_use call; reuse is resolved out of the agent loop now (recommend / route, injected or run directly). - skill_use.py docstring: reframe from "the agent invokes this from bash during solving" to the out-of-loop recommend decision (run/adapt/skip). - flights.skill.yaml: on_fail reject -> reference (match the new default). - diagrams: pipeline.svg starts from "Task batch → webwright solves them" (was "your solves") and its skill_use box is out-of-loop; interfaces.svg SOLVE TIME column no longer shows an in-loop "$ skill_use → tool output → agent observation" flow. Both PNGs regenerated from the SVGs (headless chromium).
The entry conflated the input and the solve step (and its text overflowed the box). Now two nodes: a "Task batch" box feeding a "webwright solves / a batch of runs" box, then into the gate — matching how the pipeline actually starts. PNG regenerated.
Clarified the role of `route` in skill reuse and its relationship with `recommend`. Updated descriptions for better understanding of decision-making and library growth.
…fined - checked-in flights skill: drop the unused origin_city/destination_city params; the solve logic only ever used the airport codes. CLI is now just --origin-code / --destination-code / --date (README + examples updated). - pipeline diagram: the loop entry is Task batch → webwright solves (3-card stacks, roomier padding); PNG regenerated from the SVG.
Updated the README.md to clarify the workflow and benefits of using the agent and skills in the Webwright framework. Adjusted explanations for task adaptation, skill reuse, and the distillation process.
- drop "copy-and-edit, not a clean import" — route now runs a matching skill directly via its CLI/taskspec (params in, no agent editing a copy). - drop "where the fallback lives" — that's a design note, not a limitation. - reframe "reaches too eagerly": remove the stale WebArena use-49 stat; keep the real gap — there's no cost/benefit prediction to decide when reuse pays off (a budget-aware decision is still missing).
Updated README.md to clarify skill execution and error handling.
…xample The hand-trimmed 3-param skill diverged from meta.json/replays.json (still 5) and failed test_learned_example (meta params must appear in the code). Restore the skill to what learn produced — origin_city/origin_code + destination_city/ destination_code + date — and fix the example command to pass a real city name (Seattle) alongside its code (SEA), not the code twice. Verified: runs, returns ["UA 2601","United","5:00 AM"].
Removed a note about the distiller lifting redundant parameters from the README.
… human-gated self-learning)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Web Skill Factory (
webwright.skill_factory) — a self-evolving skill factory (MVP): turn solved tasks into reusable,executable code skills, retrieve + judge them at solve time, gate what enters the library, and grow
the library incrementally. A self-evolving loop on top of Webwright's code-as-action solves:
This is the reuse + accumulation layer on top of Webwright's code-as-action solves: it consumes
the
final_script.pyevery solve already produces (plain or crafted mode — both work), accumulatesskills across tasks, judges when a prior skill applies, and improves skills as more solves arrive —
with a gate so wrong solves don't pollute the library. It complements
crafted_cli: wherecrafted_cliparameterizes a single task's script by anticipating what might vary,update.refineparameterizes across multiple verified solves — the differences actually observed between
instances become the parameters.
Modular composition (~810 lines of core code)
Ten small, single-responsibility modules — each with a stable interface and a swappable
implementation:
skill_factory/library.pySkill+Library, skills on disk (skill.py+meta.json)skill_factory/retrieve.pyskill_factory/decide.pyskill_factory/gate.pyskill_factory/update.pyrefineparameterizes + decomposes into primitivesskill_factory/llm.pyModel(no endpoint/key hardcoded)skill_factory/prompt.pywith_skill_hint)skill_factory/learn.pylearn <runs_dir>: auto-group runs into templates, gate, evolve; no manifest to writeskill_factory/__main__.pypython -m webwright.skill_factory <learn|update>dispatchertools/skill_use.pyHow it plugs in (no change to the agent loop or default config)
skill_usetool, invoked from bash likeself_reflection/image_qa:{verdict: use|adapt|skip, skill_id, source_path, how_to_reuse}.updateCLI distills a batch of gate-passed solves into aparameterized, primitive-decomposed skill:
learngroups a folder of finished runs intotemplates (one LLM call per chunk), gates them, and evolves the library — idempotent,
--dry-run:examples/learned_library/checks in the skill this produced from 3 real Google Flightssolves — five parameters lifted (origin/destination city+code, date), verified on an unseen
route three independent ways (from scratch / reuse / standalone, same answer) — with a CI
test locking it.
Validation
WebArena: 10 templates × 3 domains — reuse lifts accuracy +15pp and saves steps on held-out tasks
10 retrieve-type task templates across shopping_admin / gitlab / map. Per template: 3 train
tasks build the library (solved from scratch; only gold-verified solves are admitted), 2 held-out
tasks (unseen instances of the template — different parameter values) measure reuse. Every task is
solved both WITH the library and from scratch (BASE) — 100 solves total.
Per-task records and a reproduction driver are kept in the companion research repo and can be
shipped here on request.
Highlights:
library; net reuse-wins 7 vs 1 regression across the 20 held-out tasks.
33 steps (scratch) to 10 (reuse); a map routing task from 29 to 16.
here. (The gate is exactly as strong as its verifier — the default
self_verifyis a shapecheck only; see the README's "validation-gated" section.)
update.refinelifts per-instance differences into parametersand bakes the aggregation logic (top-n ranking, commit counting, route-time extraction) into
primitives, so unseen instances of the template solve by a direct
useof the skill.skill from the shared library (grown to 10 skills over the run), including telling apart two
near-duplicate gitlab commit-counting skills (by-date vs by-period).
evolvebatches produce 4 independent skills — new templates get added, existing skills arerefined in place (working functions kept), skills with no new traces stay byte-identical, and
zero cross-contamination between skills; held-out reuse against the mixed-built library matches
the per-template-built one.
Real website (public GitHub, read-only): the full loop end-to-end
Solve two repos from scratch ->
updatebuilds a parameterized skill -> a held-out repo is solved byreusing it (the agent calls
skill_use, verdictuse, answer correct). Reuse pays off most onmulti-step tasks where saved exploration outweighs the lookup overhead (see the WebArena numbers);
on short single-page lookups it is roughly break-even.
7 unit-test files under
tests/skill_factory/(library / gate / evolve / retrieve+decide / learn /learned-example lock / eval-snapshot lock) run in CI on every push touching the module
(
.github/workflows/skills-tests.yml).Status: a deliberately simplistic MVP
Most steps are a single LLM call (retrieve = one catalog prompt, decide = one prompt, refine =
one batched prompt) — chosen for clarity, not yet for scale/accuracy. The point is the modular
shape: each stage has a stable interface, so swapping in something stronger (embedding retrieval,
a learned ranker, WebJudge / cross-source consistency for the real-website gate) is a localized
change that does not touch the others or the agent loop.
Scope
Purely additive (zero deletions), confined to
src/webwright/skill_factory/(module + examples,including a checked-in learned skill),
src/webwright/tools/skill_use.py,tests/skill_factory/and one CI workflow. The actual implementation is ~670 lines of logic(non-blank, non-comment, across the skills module + the
skill_usetool); the rest is tests,examples, eval records, and docs. No edits to the agent loop, models, or existing configs.
Module README:
src/webwright/skill_factory/README.md.