Decompose ghost into five decentralized tools (Phase 1–5)

[nahiyankhan]

Summary

Decomposes the merged ghost-drift engine into five decentralized tools (map, expression, drift, fleet, ui). This PR carries plans, Phase 0 decisions, and Phase 1 + 2 + 3 implementation — the full decomposition.

Plans + decisions

• Five plans under docs/ideas/ — each with schema, CLI surface, recipe outline, cross-tool integration, and a reality-check section grounded in existing implementation or prior art (audited against current packages/ghost-drift and the ~/Development/ghost-fleet orchestration repo).
• docs/ideas/phase-0-decisions.md settles two blockers: per-tool skill bundles, no meta-emit; and the package layout (ghost-drift keeps its npm name and shrinks; new sibling packages for map/expression/fleet; @ghost/core is internal/private; thin top-level ghost meta-CLI deferred).

Phase 1 implementation

• @ghost/core extraction (packages/ghost-core/) — embedding math, target resolver, parameterized skill-bundle loader, shared types. ghost-drift rewires imports through it. No behavior change. Patch changeset on ghost-drift.
• ghost-map greenfield (packages/ghost-map/) — new package. Implements ghost map inventory (deterministic raw signals as JSON) and ghost map lint (zod-backed ghost.map/v1 validation + body-section partition). Hand-authored map.md fixture at the repo root. Defers describe, emit skill, and the recipe to a follow-up.
• ghost-ui canonical conventions — adds meta.expression at registry.json top level, meta.expression_dimensions on 9 representative components (palette/spacing/typography/surfaces vocabulary), canonical map.md, README convention doc, draft .github/workflows/ghost-ci.yml (gated off; captures intended shape).

Phase 2 implementation

• ghost-expression extraction (packages/ghost-expression/) — new package carrying the canonical artifact and emit verbs. Moves core/expression/, core/context/, emit-command.ts, the profile/schema recipes from ghost-drift. New CLI: lint, describe, diff (new verb wrapping the existing diffExpressions() library function), emit (review-command | context-bundle | skill). SKILL.md added. Profile recipe rewritten to consume map.md as optional input with inline-discovery fallback when missing.
• ghost-drift trim — drops lint, describe, emit review-command, emit context-bundle from CLI (stub commands print migration messages pointing at ghost-expression and exit 2). Drift retains compare, ack, track, diverge, emit skill. New remediate.md recipe added; discover.md and generate.md dropped (per Phase 0 decision). SKILL.md updated. Library core/index.ts proxies all moved exports for one major-version cycle as a deprecation shim.
• Dependency graph: @ghost/core ← ghost-expression ← ghost-drift. No circular deps.
• Changesets: minor for ghost-expression (initial), major for ghost-drift (CLI surface change).

Phase 3 implementation

• ghost-fleet greenfield (packages/ghost-fleet/) — read-only world model over a directory of (map.md, expression.md) members. Three CLI verbs: members (lists members + freshness), view (emits fleet.md + fleet.json to <dir>/reports/ with pairwise distances, group-by tables across five axes from map.md, tracks-graph from per-member .ghost-sync.json), emit skill.
• ghost.fleet/v1 schema — pairwise distances array (not matrix); clusters live in body narrative, not frontmatter (matches prior-art's narrow JSON output). Body has three required skeleton sections (## World shape, ## Cohorts, ## Tracks) the skill recipe fills.
• Skill bundle ships target.md only (monolithic-target reasoning). module.md / rollup.md deferred until the modular profile pathway lands in expression.
• Fixture: test/fixtures/small-fleet/ with three members (cash-web, cash-android, ghost-ui) covering two platforms and two registry states; cash-web declares it tracks ghost-ui to exercise the tracks-graph.
• Composite math relocation deferred: fleet uses compareExpressions from @ghost/core directly for pairwise; centroid + spread + clustering remain in ghost-drift/core/evolution/composite.ts. Defensible — clusters are skill-layer per the schema, so fleet's CLI doesn't need them. Move can happen later if a deterministic clustering verb is added.

Test plan

• pnpm install clean
• pnpm build clean
• pnpm test — 219/219 across 25 files (was 196 pre-Phase 2, 201 pre-Phase 3; +18 fleet tests this phase)
• pnpm check — biome + typecheck + file-sizes + docs frontmatter + CLI manifest
• Lefthook pre-commit on every commit
• Smoke: node packages/ghost-expression/dist/bin.js lint packages/ghost-ui/expression.md → 0/0/0
• Smoke: node packages/ghost-drift/dist/bin.js compare --help → expected output
• Smoke: node packages/ghost-drift/dist/bin.js lint → migration message
• Smoke: node packages/ghost-fleet/dist/bin.js members <fixture> → 3 members listed
• Smoke: node packages/ghost-fleet/dist/bin.js view <fixture> → fleet.md + fleet.json emitted
• Manual review: canonical expression.md and map.md fixtures for ghost-ui
• Manual review: refactored profile.md recipe (map-aware fallback)
• Manual review: new remediate.md recipe shape
• Manual review: drift's library backcompat shim — confirm one major cycle is the right deprecation window
• Manual review: ghost.fleet/v1 schema + small-fleet fixture

Notes for review

• Six Merge ... (Phase N) / feat / refactor commits keep each worktree's contribution separable for cherry-pick if the PR needs splitting.
• core/evolution/vector.ts moved with embedding to @ghost/core (would have created a @ghost/core → ghost-drift → @ghost/core cycle otherwise). Re-exported through core/evolution/index.ts so drift's evolution surface is identical.
• discover.md and generate.md recipes are dropped — not migrated to any package per Phase 0 decision.
• Composite math (centroid, spread, clustering) stays in drift for now. Fleet's pairwise computation uses the lower-level compareExpressions from @ghost/core directly. Move to @ghost/core deferred until a tool needs it.
• Explicitly deferred to next milestones (with reasoning in plans): --by-component flag on drift compare, .ghost-sync.json per-component schema, top-level ghost meta-CLI dispatcher, ghost-map describe + skill recipe, ghost-fleet tracks / temporal / --include-modules / --groupby axis filtering, multi-product expression mode: parameter on profile recipe, fleet's module.md / rollup.md skill fragments.

🤖 Generated with Claude Code

[nahiyankhan]

Phase 4a — Tier 1+2 detection fixes (commit 5f682d5, merge e7e2de7)

Seven fixes folded into one commit:

Tier 1 (hard blockers):

• docs/expression-format.md reconciled with the strict FrontmatterSchema — evidence: removed from frontmatter examples, schema: root key removed, schema generation called out as v5
• profile.md recipe: design_system.location → design_system.paths
• Inventory manifest detection: added Bazel (WORKSPACE/MODULE.bazel/BUILD.bazel), Maven (pom.xml), Gemfile{,.lock}, *.gemspec, mix.exs, composer.json, Package.resolved, setup.py. Closes the Bazel-repo hard-fail.

Tier 2 (detection gaps):

• Language-histogram-driven platform inference (Swift>40%+SPM/Xcode/Bazel → ios; Kotlin/Java + Gradle + AndroidManifest.xml → android; Dart+pubspec.yaml → flutter; multi-platform → mixed). Edge cases (KMP, RN, MAUI, Tauri) documented as future work.
• Token-directory signal matchers (tokens/, design-tokens/, theme/, themes/)
• SKIP_DIRS extended (universal patterns: venv, .venv, dist-*, bazel-*, .next, .nuxt, .svelte-kit, .turbo, .fleet)
• candidate_config_files ancestor-aware scoring (DS-ancestor paths sort first; Code/DesignSystem/Color+Brand.swift > features/banking/Color+Banking.swift)

Test fixtures added (tiny marker-file repos): bazel-repo, ios-spm-repo, android-gradle-repo, flutter-repo, token-pipeline-repo, python-venv-repo. Inventory tests grew from 3 → 18.

Verification: 234/234 tests pass (was 219), all checks green, lefthook pre-commit clean.

Notable bonus fix: scripts/emit-expression-schema.mjs was importing from packages/ghost-drift/dist/... — broken since the Phase 2 schema migration. Now imports from packages/ghost-expression/dist/....

[nahiyankhan]

Phase 4b — schema enrichment (commit aeb2281, merge 25ad774)

Seven coupled changes across ghost-map and ghost-expression (with cross-package fallout into @ghost/core and ghost-fleet):

Tier 2 remaining:

• Workspace expansion in inventory. Honors package.json:workspaces (array + {packages: []} shape) plus conventional apps/, packages/, libs/, common/ one-level descent. Deduped by absolute path. Root entries stay basenames; nested entries are POSIX-relative. Fixture proves a 6-manifest workspace surfaces correctly.
• platform: T | T[] schema accepts both shapes. mixed kept for backcompat; [ios, android, web] preferred for clarity.
• build_system: T | T[] — same shape. Enum extended with style-dictionary, maven, sbt, cmake. Cargo was already present.
• New build_system_hints inventory field alongside platform_hints — separates platform from build-system signals (cleaner than my brief asked for).

Tier 3 enrichment:

• design_system.token_source: inline | external | mixed + optional design_system.upstream (free-form: npm package, SPM ref, sibling path).
• design_system.derived_files alongside entry_files. Token pipelines can declare both source-of-truth and built artifacts. entry_files now optional; new lint rule warns if both missing.
• Three new lint rules: design-system-files-missing (warn), design-system-upstream-missing (warn when external without upstream), design-system-upstream-stranded (info when upstream set without external).
• unused-palette lint relaxed — now counts citations in roles[].evidence and roles[].tokens.palette.{background,foreground,border}. Confirmed sufficient: ghost-ui/expression.md goes from 8 INFO to 0 without the opt-out flag.
• shadowComplexity: none → deliberate-none (only deliberate breaking change). Cross-package fallout: @ghost/core types + embedding union updated; tests cover schema rejection of legacy value.

Cross-package fallout the agent caught:

• ghost-fleet/compute.ts updated to handle array-shape platform/build_system (cross-tabulates per value, formats multi-value cells as comma-separated).
• @ghost/core shadowComplexity union updated.
• Skill bundle assets, docs/expression-format.md, docs/ideas/ghost-map.md schema table all updated to reflect new shapes.

Verification: 254/254 tests pass (was 234, +20), all checks green, lefthook pre-commit clean. Both canonical fixtures (packages/ghost-ui/{map.md,expression.md}) lint at 0/0/0.

Changeset: ghost-expression: minor. Pre-publish status (0.0.0); the breaking shadowComplexity rename is called out in the changeset body but doesn't warrant major given the unpublished state.

[nahiyankhan]

Phase 4c — profile recipe branching (commit 2758723, merge 7515e69)

Single recipe-doc edit. Profile recipe now branches by detected repo kind:

New Step 1.5 (detection, first-match-wins):

1. token_source: external → consumer mode
2. style-dictionary framework or tokens/ dir without registry → token-pipeline mode
3. registry.path set or styling implies code/CSS → ui-library mode (default)
4. Multi-platform array → ui-library with coarser feature_areas + native module-suffix preference (*UI / *View / *Screen; skip *Fakes / *Mocks / *Tests)
5. Tie-breaker: token-pipeline when entry_files are YAML/JSON graphs, ui-library when CSS/code

Branched sampling (Step 3):

• ui-library: 6–10 component files, feature_areas = registry categories
• token-pipeline: 3–5 component YAMLs walked through layer chain (component → semantic → base), feature_areas = token-architecture layers or platform sinks, light/dark surfaced together
• consumer: 6–10 product UI files, record most-frequent upstream slugs + override patterns; don't fabricate hex

Phase 0 module-suffix decision reaffirmed — texture stays in Topology prose, surfaces in the recipe as a one-line sampling heuristic for native repos.

Recipe size: 112 → 108 lines (under the 150 cap).

Verification: 254/254 tests pass, all checks green, both ghost-ui canonical fixtures lint at 0/0/0.

Changeset: ghost-expression: patch (recipe-only, no schema or behavior changes).

[nahiyankhan]

Phase 5a + 5b — re-validation findings (commits c0601b4, 89babaa)

Two commits closing the gaps surfaced by the second-round validation pass.

Phase 5a — Tier A bug fixes

• schema.md skill-bundle reference reconciled. The Phase 4a fix to docs/expression-format.md missed the LLM-facing condensed copy in the skill bundle — the file the recipe directs the agent to read. Removed decisions[].evidence from frontmatter examples, added body-side **Evidence:** examples, fixed the partition table.
• profile.md Step 5 carries explicit "evidence body-only" warning.
• unused-palette slug-binding propagation works. Added collectSlugBoundHexes() that walks roles[].tokens.palette.<slot> references through resolveTokenReference and marks the resolved hex as cited. (Existing role-evidence string handling from 4b was correct — only this propagation was missing.)
• platform_hints no longer leaks build-system signals. derivePlatformHints now emits only Platform-enum values, with a defensive filter at the end. Bazel/Ruby/Python/Rust/Go/Elixir/PHP/JVM all dropped from platform_hints — they continue to surface in build_system_hints as appropriate.
• New fixture: bazel-ruby-ios-repo modeling the leak case.

Phase 5b — Tier B schema gaps for real-world variety

• build_system enum extended with vite, webpack, parcel, rollup, turbopack, esbuild, nx, turbo. New BUILD_TOOL_MATCHERS surface them in build_system_hints.
• design_system.upstream accepts string | string[]. Multi-upstream consumers (token package + components + icons + glue) can list them honestly.
• roles[].tokens.palette slot vocabulary opened. Was a strict 3-key object (background/foreground/border); now z.record(z.string(), z.string()). Conventional keys (background, foreground, surface, border, accent, muted, link) documented in schema.md and expression-format.md so agents reach for them first.
• broken-role-reference accepts nested external token refs. Style-Dictionary-style refs like {base.color.brand.x.light} no longer rejected. New isExternalTokenReference heuristic recognizes heads (base, core, semantic, component, tokens, ref, sys) or 4+ dotted segments. Local palette.* refs still validate strictly.
• New fixtures: vite-nx-repo, multi-upstream-repo.

Verification: 262 tests pass (was 254, +8 from new fixtures). All checks green. Both ghost-ui canonical fixtures lint at 0/0/0. Lefthook pre-commit clean on both commits.

Note: The brief's claim that 4b's role-evidence string handling didn't trigger turned out to be wrong — that part actually worked. The genuine gap was only slug-binding propagation, which is now fixed. The agent verified the existing role-evidence test path before changing anything.