Repository navigation
Releases: christophhdesign/open-design-system-bench
Release list
Release 1.3.0: UI gallery, opt-in hard tasks, gateway streaming.
Release 1.3.0: UI gallery, opt-in hard tasks, gateway streaming.
Field feedback from the talk, plus what this week's test runs turned up.
UI gallery. gallery [<runDir>...] renders a page with a screenshot of
what every ok cell rendered: each workspace is built with vite and
captured with headless Chrome into screenshot.png next to grades.json,
cached per cell, with build failures recorded in screenshot-error.txt and
shown on the card. Rows per context level, cards per rep with score,
gate, turns, cost and a dimensions drill-down; task picker, gate filter,
context toggles, size slider; several runs get a run picker. Chrome,
Chromium or Edge on the machine, or CHROME_PATH; no new dependency. A
ds-bench-gallery skill wraps the command. Console errors Chrome reports
while rendering are captured next to the screenshot when present.
Hard tasks, opt-in. tasks/hard/ holds bulk-member-actions,
async-lookup-states, inline-profile-editing and keyboard-action-finder,
each combining several patterns with state, async or keyboard work.
Prompts pass the leak lint against nine production catalogs. A profile's
"*" still means the ten starter tasks, so existing baselines are
unchanged; --hard or "hard": true in a profile adds them, and the new
hard profile is exactly that across all three contexts.
Gateways. "stream": true on an openai-kind provider streams the
completion (SSE) and reassembles it client-side, for gateways that
buffer whole completions and time out on long generations (HTTP 408
after 200 s on one gateway, which single-shot cells on the hard tasks
exceed). extractJsonPayload now skips a stray { on its own line before
the real object, which one open-weights model produced on three of four
hard-task cells; those cells are graded instead of counted as errors.
pricing-catalog.json is documented and gitignored: it supplies each
model's real max_tokens and prices, and a 400 about max_tokens now hints
at it. Two tests that assumed no catalog at the package root now use an
unpriceable model name.
Docs: README what's new for v1.3, profile table, gallery and catalog
sections; CHANGELOG 1.3.0; AGENTS.md map. package.json version 1.3.0.
1.2 Isolate agent cells, follow solution-style tsconfigs, grade style sheets.
Field feedback from running the audit and smoke tests against a Vite-based
React design system.
Bare cells were not bare. The claude-code adapter loaded the operator's
~/.claude/CLAUDE.md, user skills, plugins, hooks, effort level and
auto-memory, and, since runs/ sits inside this repo, the bench's own
CLAUDE.md and AGENTS.md through Claude Code's parent-directory walk. Cells
now pass --setting-sources project plus --settings with autoMemoryEnabled
off and claudeMdExcludes for every ancestor's *.md and .claude/, so the
workspace's own CLAUDE.md and .claude/skills are the only context left.
--restricted, --safe-mode and --bare were rejected: each also drops those
workspace files, which would collapse agents-md and skill into bare. The
judge needs no context and now runs with --safe-mode. Verified against a
request-capturing mock endpoint. Earlier runs carried this context, so
comparisons against older baselines will shift.
docgen follows a solution-style root tsconfig ("files": [] plus
references, the Vite default) to the reference that includes
componentsSrc, and warns with the file it used. A new optional
SystemConfig.tsconfig field overrides the lookup. Before, the root's empty
options left path aliases unresolved and the catalog came back with almost
no props.
1.1: Merge pull request #7 from christophhdesign/fix-false-positives
v1.1
The static audit and extract commands were field-tested against eight production
design systems (Chakra UI, Mantine, MUI, Ant Design, Carbon, Primer, Base UI, Radix
Primitives) and every wrong finding was traced to its mechanism and fixed. Audit
scores from this release are not comparable to earlier ones: several checks measure
more, lie less, and score differently.
Extraction: works on real-world monorepos now
- tsconfig.json is resolved upward from componentsSrc to the system root instead of
assumed one level up; unknown compiler options from newer TypeScript versions
(e.g. TS 6.0'sstableTypeOrdering) are warned about and tolerated instead of
crashing. - The barrel walker follows
export *chains recursively, resolves NodeNext
specifiers (./dir/index.js,./Button.js), acceptsindex.js/.jsx/.d.ts
barrels, follows re-exports that resolve outside componentsSrc but inside the
system root, and keeps parent-side default aliases
(export { default as X } from './X') when a resolvableexport *names the
same dir. - Multi-system extract isolates failures per system: one broken system no longer
kills the rest, and the command exits nonzero with a summary when any failed. - Props are split into own (full metadata) and inherited (name-only, new
CatalogExport.inheritedProps): declared outside componentsSrc or in a shared
type feeding 20+ exports.allPropsByExportstill carries own plus inherited, so
grading is unchanged. A prop-heavy real-world catalog dropped from 130 MB to
under 25 MB. - The extract summary line now distinguishes documented exports from barrel
exports, and warns when a catalog exceeds 10 MB.
Audit: fewer false verdicts, evidence everywhere
- Workspace-aware scanning (package.json workspaces and pnpm-workspace.yaml, real
globstar semantics): MCP servers, codemod packages, and changelogs in sibling
workspace packages are found and named..claude/builder tooling is reported. - The MCP hint no longer false-positives on unrelated dependency names, never
awards points silently, and recognizes.well-known/mcp. - Export hygiene accepts esm/cjs/lib build outputs (not just dist/), understands
exports-map"./*"wildcards, and reads NodeNext barrel specifiers. - Dark-mode detection matches any
data-*-theme/data-*-color-schemeattribute,
.darkselectors, andcolor-scheme:declarations. - Docs greppability matches compound names (
Accordion.ItemBody), labels its two
scoring modes, and caps the no-catalog volume fallback at 40 so it can never
outscore measured coverage. - Doc freshness uses git commit history instead of file mtimes (meaningless on
fresh clones) and reports "unmeasured" outside a git checkout. - Zero-prop exports are named, with an extraction-suspect warning at 30%+; an
empty extracted catalog is treated as unmeasured with a loud warning, never
scored as a bad system. - Changelog detection accepts CHANGELOG*.md variants and
.changeset/.
New: hosted-surface probes (opt-in) and token partial credit
SystemConfig.docsUrlenables HTTP probes for/llms.txt,/llms-full.txt,
/mcp/index.json, and/registry.json. Findings carry the URL that proves
them; a 404 is measured absence, a timeout is unmeasured. Without docsUrl the
audit touches no network;audit --offlineforces that even with one. Hosted
llms.txt content feeds per-component coverage when small enough, and every llms
artifact gets a context-cost finding (warning above 1 MB).- Token machine-readability awards capped partial credit (max 45) for pre-build
SCSS custom properties and TS-defined token systems (defineTokens,
defineSemanticTokens,createTheme,semanticTokensobjects), with fix text
pointing at compiled CSS or DTCG JSON for full credit.
Also
initprobes candidate componentsSrc dirs and ranks them by real export counts
(covers monorepo layouts, root-levelcomponents/, and JS-only barrels); it
asks for docsUrl.doctorchecks docsUrl reachability. AbsolutedataDirpaths resolve
correctly. The test suite grew from 118 to 200 tests, all offline.
Known limitations, tracked for the next releases: aggregate packages that
re-export via package specifiers (the Radix shape) extract empty and are reported
as unmeasured; docgen still returns no props for factory/polymorphic component
patterns (loudly flagged, not yet solved).
1.0
Merge pull request #6 from christophhdesign/fix-false-positives - Further bug fixes after more test runs