Skip to content

Releases: christophhdesign/open-design-system-bench

Release 1.3.0: UI gallery, opt-in hard tasks, gateway streaming.

Choose a tag to compare

@christophhdesign christophhdesign released this 05 Oct 16:48
fbdbe8e

Release 1.3.0: UI gallery, opt-in hard tasks, gateway streaming.

Field feedback from the talk, plus what this week's test runs turned up.

UI gallery. gallery [<runDir>...] renders a page with a screenshot of
what every ok cell rendered: each workspace is built with vite and
captured with headless Chrome into screenshot.png next to grades.json,
cached per cell, with build failures recorded in screenshot-error.txt and
shown on the card. Rows per context level, cards per rep with score,
gate, turns, cost and a dimensions drill-down; task picker, gate filter,
context toggles, size slider; several runs get a run picker. Chrome,
Chromium or Edge on the machine, or CHROME_PATH; no new dependency. A
ds-bench-gallery skill wraps the command. Console errors Chrome reports
while rendering are captured next to the screenshot when present.

Hard tasks, opt-in. tasks/hard/ holds bulk-member-actions,
async-lookup-states, inline-profile-editing and keyboard-action-finder,
each combining several patterns with state, async or keyboard work.
Prompts pass the leak lint against nine production catalogs. A profile's
"*" still means the ten starter tasks, so existing baselines are
unchanged; --hard or "hard": true in a profile adds them, and the new
hard profile is exactly that across all three contexts.

Gateways. "stream": true on an openai-kind provider streams the
completion (SSE) and reassembles it client-side, for gateways that
buffer whole completions and time out on long generations (HTTP 408
after 200 s on one gateway, which single-shot cells on the hard tasks
exceed). extractJsonPayload now skips a stray { on its own line before
the real object, which one open-weights model produced on three of four
hard-task cells; those cells are graded instead of counted as errors.
pricing-catalog.json is documented and gitignored: it supplies each
model's real max_tokens and prices, and a 400 about max_tokens now hints
at it. Two tests that assumed no catalog at the package root now use an
unpriceable model name.

Docs: README what's new for v1.3, profile table, gallery and catalog
sections; CHANGELOG 1.3.0; AGENTS.md map. package.json version 1.3.0.

1.2 Isolate agent cells, follow solution-style tsconfigs, grade style sheets.

Choose a tag to compare

@christophhdesign christophhdesign released this 05 Oct 10:42
e4bd4c7

Field feedback from running the audit and smoke tests against a Vite-based
React design system.

Bare cells were not bare. The claude-code adapter loaded the operator's
~/.claude/CLAUDE.md, user skills, plugins, hooks, effort level and
auto-memory, and, since runs/ sits inside this repo, the bench's own
CLAUDE.md and AGENTS.md through Claude Code's parent-directory walk. Cells
now pass --setting-sources project plus --settings with autoMemoryEnabled
off and claudeMdExcludes for every ancestor's *.md and .claude/, so the
workspace's own CLAUDE.md and .claude/skills are the only context left.
--restricted, --safe-mode and --bare were rejected: each also drops those
workspace files, which would collapse agents-md and skill into bare. The
judge needs no context and now runs with --safe-mode. Verified against a
request-capturing mock endpoint. Earlier runs carried this context, so
comparisons against older baselines will shift.

docgen follows a solution-style root tsconfig ("files": [] plus
references, the Vite default) to the reference that includes
componentsSrc, and warns with the file it used. A new optional
SystemConfig.tsconfig field overrides the lookup. Before, the root's empty
options left path aliases unresolved and the catalog came back with almost
no props.

1.1: Merge pull request #7 from christophhdesign/fix-false-positives

Choose a tag to compare

@christophhdesign christophhdesign released this 28 Aug 09:14
1339e7e

v1.1

The static audit and extract commands were field-tested against eight production
design systems (Chakra UI, Mantine, MUI, Ant Design, Carbon, Primer, Base UI, Radix
Primitives) and every wrong finding was traced to its mechanism and fixed. Audit
scores from this release are not comparable to earlier ones: several checks measure
more, lie less, and score differently.

Extraction: works on real-world monorepos now

  • tsconfig.json is resolved upward from componentsSrc to the system root instead of
    assumed one level up; unknown compiler options from newer TypeScript versions
    (e.g. TS 6.0's stableTypeOrdering) are warned about and tolerated instead of
    crashing.
  • The barrel walker follows export * chains recursively, resolves NodeNext
    specifiers (./dir/index.js, ./Button.js), accepts index.js/.jsx/.d.ts
    barrels, follows re-exports that resolve outside componentsSrc but inside the
    system root, and keeps parent-side default aliases
    (export { default as X } from './X') when a resolvable export * names the
    same dir.
  • Multi-system extract isolates failures per system: one broken system no longer
    kills the rest, and the command exits nonzero with a summary when any failed.
  • Props are split into own (full metadata) and inherited (name-only, new
    CatalogExport.inheritedProps): declared outside componentsSrc or in a shared
    type feeding 20+ exports. allPropsByExport still carries own plus inherited, so
    grading is unchanged. A prop-heavy real-world catalog dropped from 130 MB to
    under 25 MB.
  • The extract summary line now distinguishes documented exports from barrel
    exports, and warns when a catalog exceeds 10 MB.

Audit: fewer false verdicts, evidence everywhere

  • Workspace-aware scanning (package.json workspaces and pnpm-workspace.yaml, real
    globstar semantics): MCP servers, codemod packages, and changelogs in sibling
    workspace packages are found and named. .claude/ builder tooling is reported.
  • The MCP hint no longer false-positives on unrelated dependency names, never
    awards points silently, and recognizes .well-known/mcp.
  • Export hygiene accepts esm/cjs/lib build outputs (not just dist/), understands
    exports-map "./*" wildcards, and reads NodeNext barrel specifiers.
  • Dark-mode detection matches any data-*-theme/data-*-color-scheme attribute,
    .dark selectors, and color-scheme: declarations.
  • Docs greppability matches compound names (Accordion.ItemBody), labels its two
    scoring modes, and caps the no-catalog volume fallback at 40 so it can never
    outscore measured coverage.
  • Doc freshness uses git commit history instead of file mtimes (meaningless on
    fresh clones) and reports "unmeasured" outside a git checkout.
  • Zero-prop exports are named, with an extraction-suspect warning at 30%+; an
    empty extracted catalog is treated as unmeasured with a loud warning, never
    scored as a bad system.
  • Changelog detection accepts CHANGELOG*.md variants and .changeset/.

New: hosted-surface probes (opt-in) and token partial credit

  • SystemConfig.docsUrl enables HTTP probes for /llms.txt, /llms-full.txt,
    /mcp/index.json, and /registry.json. Findings carry the URL that proves
    them; a 404 is measured absence, a timeout is unmeasured. Without docsUrl the
    audit touches no network; audit --offline forces that even with one. Hosted
    llms.txt content feeds per-component coverage when small enough, and every llms
    artifact gets a context-cost finding (warning above 1 MB).
  • Token machine-readability awards capped partial credit (max 45) for pre-build
    SCSS custom properties and TS-defined token systems (defineTokens,
    defineSemanticTokens, createTheme, semanticTokens objects), with fix text
    pointing at compiled CSS or DTCG JSON for full credit.

Also

  • init probes candidate componentsSrc dirs and ranks them by real export counts
    (covers monorepo layouts, root-level components/, and JS-only barrels); it
    asks for docsUrl.
  • doctor checks docsUrl reachability. Absolute dataDir paths resolve
    correctly. The test suite grew from 118 to 200 tests, all offline.

Known limitations, tracked for the next releases: aggregate packages that
re-export via package specifiers (the Radix shape) extract empty and are reported
as unmeasured; docgen still returns no props for factory/polymorphic component
patterns (loudly flagged, not yet solved).

1.0

1.0

Choose a tag to compare

@christophhdesign christophhdesign released this 28 Aug 06:52
5983367
Merge pull request #6 from christophhdesign/fix-false-positives

- Further bug fixes after more test runs