-
Notifications
You must be signed in to change notification settings - Fork 4
feat(webllm): rewrite-quality eval harness (#65) #149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| <!doctype html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="UTF-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | ||
| <title>resumelint — rewrite eval (dev only)</title> | ||
| <meta | ||
| name="description" | ||
| content="Dev-only eval harness for rewrite quality (issue #65). Not part of the production bundle." | ||
| /> | ||
| <meta name="color-scheme" content="light dark" /> | ||
| <!-- | ||
| Dev-only tool page; not subject to the design-token policy that | ||
| governs feature code under src/components/**. CSS keeps to system | ||
| colors (Canvas / CanvasText) and a small set of inline rules so | ||
| the page renders standalone without loading the app's design system. | ||
| --> | ||
| <style> | ||
| :root { | ||
| color-scheme: light dark; | ||
| font-family: ui-sans-serif, system-ui, sans-serif; | ||
| } | ||
| body { | ||
| max-width: 880px; | ||
| margin: 2rem auto; | ||
| padding: 0 1rem; | ||
| line-height: 1.5; | ||
| color: CanvasText; | ||
| background: Canvas; | ||
| } | ||
| h1 { margin-bottom: 0.25rem; } | ||
| .subtitle { color: GrayText; margin-top: 0; } | ||
| button { | ||
| padding: 0.6rem 1.2rem; | ||
| font-size: 1rem; | ||
| cursor: pointer; | ||
| border-radius: 6px; | ||
| } | ||
| button[disabled] { opacity: 0.5; cursor: not-allowed; } | ||
| #status { font-weight: 600; margin: 1rem 0 0.25rem; } | ||
| #progress { font-family: ui-monospace, monospace; color: GrayText; } | ||
| #log { | ||
| background: color-mix(in srgb, CanvasText 92%, Canvas); | ||
| color: Canvas; | ||
| padding: 0.75rem; | ||
| border-radius: 6px; | ||
| font-family: ui-monospace, monospace; | ||
| font-size: 0.85rem; | ||
| height: 320px; | ||
| overflow-y: auto; | ||
| white-space: pre-wrap; | ||
| margin-top: 1rem; | ||
| } | ||
| .downloads { | ||
| margin-top: 1rem; | ||
| display: flex; | ||
| gap: 0.75rem; | ||
| } | ||
| .downloads a[hidden] { display: none; } | ||
| .downloads a { | ||
| padding: 0.5rem 0.9rem; | ||
| border: 1px solid currentColor; | ||
| border-radius: 6px; | ||
| text-decoration: none; | ||
| } | ||
| .note { | ||
| font-size: 0.9rem; | ||
| color: GrayText; | ||
| margin-top: 1.5rem; | ||
| } | ||
| </style> | ||
| </head> | ||
| <body> | ||
| <h1>Rewrite eval harness</h1> | ||
| <p class="subtitle"> | ||
| Issue #65 · dev-only · WebGPU required · not bundled into production. | ||
| </p> | ||
| <p> | ||
| Runs the selected model against every prompt variant against every | ||
| fixture under <code>tests/fixtures/rewrite/</code>. Deterministic | ||
| rubric only — no judge model is loaded. | ||
| </p> | ||
| <p> | ||
| One model per tab on purpose: cycling multiple multi-GB models in a | ||
| single browser tab is fragile on consumer GPUs (eviction-then-reload | ||
| can OOM). Run each model in a fresh tab, download its report, then | ||
| open a new tab for the next one. | ||
| </p> | ||
| <p> | ||
| <label for="model">Model: </label> | ||
| <select id="model"></select> | ||
| | ||
| <button id="run" type="button">Run eval</button> | ||
| </p> | ||
| <p id="status">Idle. Click “Run eval” to start.</p> | ||
| <p id="progress"> </p> | ||
| <div class="downloads"> | ||
| <a id="download-json" hidden>Download JSON report</a> | ||
| <a id="download-md" hidden>Download Markdown report</a> | ||
| </div> | ||
| <pre id="log"></pre> | ||
| <p class="note"> | ||
| The downloaded report goes under | ||
| <code>tests/fixtures/rewrite/reports/</code> when you commit a fresh | ||
| run. See <code>src/lib/webllm/eval/README.md</code> for the | ||
| report-commit workflow. | ||
| </p> | ||
| <script type="module" src="/src/lib/webllm/eval/run-eval-browser.ts"></script> | ||
| </body> | ||
| </html> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| # Rewrite-quality eval harness | ||
|
|
||
| Phase 3 of the in-browser AI rewrite epic (issue #65). Scores | ||
| section-rewrite outputs against a deterministic rubric so the default | ||
| model + prompt are picked from measurement rather than vibes. | ||
|
|
||
| ## Layout | ||
|
|
||
| ``` | ||
| src/lib/webllm/eval/ | ||
| ├── types.ts # FixtureKind, RubricResult, EvalReport, RewriteFn | ||
| ├── verbs.ts # curated action-verb set (superset of scorer's) | ||
| ├── fixtures.ts # loads + validates JSON fixtures | ||
| ├── rubric.ts # the six deterministic criteria | ||
| ├── prompt-variants.ts # the shipped prompt + experimental variants | ||
| ├── runner.ts # iterates (model × variant × fixture) | ||
| ├── report.ts # JSON + Markdown formatters | ||
| └── run-eval-browser.ts # browser entry — wires real WebLLM engine | ||
| ``` | ||
|
|
||
| Fixtures live under `tests/fixtures/rewrite/`; reports get committed to | ||
| `tests/fixtures/rewrite/reports/`. | ||
|
|
||
| ## Two execution legs | ||
|
|
||
| ### 1. Scoring leg (CI) | ||
|
|
||
| Pure scoring logic — rubric, runner, formatters, fixture loading — all | ||
| unit-tested under `*.test.ts` siblings. Runs in the default | ||
| `npm run test` and is exercised on every PR via the existing CI gate. | ||
| No model, no WebGPU, no network. | ||
|
|
||
| ### 2. Inference leg (local, WebGPU) | ||
|
|
||
| Real models run only in a browser. The entry point is the dev-only | ||
| `eval-rewrite.html` page at the project root: | ||
|
|
||
| ```sh | ||
| npm run eval:rewrite | ||
| # opens http://localhost:5173/resumelint/eval-rewrite.html | ||
| ``` | ||
|
|
||
| **One model per tab.** The page asks you to pick a model from the | ||
| dropdown, then click **Run eval** — it loads that model only, runs every | ||
| prompt variant against every fixture, scores with the rubric, and | ||
| exposes JSON + Markdown report downloads. To compare another model, | ||
| open a fresh tab (or refresh) and pick a different one. | ||
|
|
||
| This is intentional: cycling several multi-GB models in a single tab | ||
| kept crashing Chrome on consumer GPUs during the WebGPU | ||
| eviction-then-reload path. Closing and reopening the tab between | ||
| models reclaims VRAM cleanly. The downside is the maintainer commits | ||
| one report file per model and reviewers compare them side-by-side — | ||
| still cheap. | ||
|
|
||
| Each downloaded report includes the model slug in the filename | ||
| (`eval-rewrite-qwen2-5-1-5b-…-{timestamp}.{json,md}`) so the three | ||
| per-model files coexist under `tests/fixtures/rewrite/reports/` without | ||
| collision. Reports are append-only — never overwrite a prior run. | ||
|
|
||
| `eval-rewrite.html` is NOT included in `build.rollupOptions.input`, so | ||
| the production bundle is unaffected. | ||
|
|
||
| ## Reading the report | ||
|
|
||
| The Markdown report leads with a per-`(model, variant)` aggregate row. | ||
| Six rates (numbers / one-line / verb / length / no-preamble / dedup) and | ||
| the equal-weight composite `Aggregate` column drive the model choice. | ||
| Per-cell records below the aggregate let you trace a failure to a | ||
| specific fixture. | ||
|
|
||
| The dedup column is `—` for non-redundant fixtures (the criterion | ||
| doesn't apply); the aggregate's dedup rate is computed over `redundant` | ||
| fixtures only. | ||
|
|
||
| The judge column is `—` until the optional LLM-judge gate is enabled. | ||
| That path is flag-plumbed (`runEval({ judgeEnabled })`) but the | ||
| implementation is intentionally stubbed — coherence judging is a follow-up. | ||
|
|
||
| ## Adding a fixture | ||
|
|
||
| Drop a JSON file under `tests/fixtures/rewrite/` with this shape: | ||
|
|
||
| ```json | ||
| { | ||
| "id": "kebab-case-id", | ||
| "kind": "weak | strong | numeric | redundant", | ||
| "description": "What this fixture stresses, for the report's prose.", | ||
| "bullets": ["...", "..."] | ||
| } | ||
| ``` | ||
|
|
||
| Then append an `import` + entry in `fixtures.ts::REWRITE_FIXTURES`. | ||
| `parseFixture` validates shape at module load — a malformed fixture | ||
| throws with a precise pointer before any eval runs. | ||
|
|
||
| **PII policy still applies.** Bullet fixtures are persona-free by | ||
| construction (no contact info), but keep employer names, dates, and | ||
| résumé details synthetic. The repo is public. | ||
|
|
||
| ## Adding a prompt variant | ||
|
|
||
| Append to `prompt-variants.ts::PROMPT_VARIANTS`. The runner enumerates | ||
| the array; the browser entry picks all of them up automatically. Keep | ||
| deltas small — one or two rule changes per variant — so a regression in | ||
| any one criterion traces cleanly to the prompt change. | ||
|
|
||
| ## Choosing a default model | ||
|
|
||
| The aggregate's `Aggregate` column is the equal-weight mean of the | ||
| deterministic rates. If two models tie within ~3 points, prefer the | ||
| smaller / Apache-2.0 one — the eval is a measurement floor, not the only | ||
| input (license, download size, and consent friction matter for the | ||
| shipped default). |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.