feat(cli): convention-based docs discovery pipeline

[amondnet]

Closes #46

Summary

Replaces the registry-first resolver with a convention-based discovery pipeline that runs BEFORE the central registry lookup for npm: ecosystem specs. Most popular OSS libraries now resolve automatically with zero registry curation, and TanStack Intent-format packages wire into AGENTS.md's native marker block instead of being copied into .ask/docs/.

The central registry is not removed — it is demoted to a fallback for libraries with non-conventional layouts or alias mappings.

Highlights

New discovery module (packages/cli/src/discovery/)

• local-ask — package.json.ask.docsPath opt-in (library-author contract)
• local-intent — wraps @tanstack/intent's findSkillFiles / parseFrontmatter with zod runtime validation
• local-conventions — dist/docs → docs → README.md fallback with a warning
• repo-conventions — conventional-path scanner for downloaded github repo trees
• Quality scorer — >=3 markdown files OR >=4 KiB threshold with a LICENSE / CHANGELOG / CONTRIBUTING exclusion filter (SC-3 guard against noise-only repos)

Dual AGENTS.md marker blocks

• Existing <!-- BEGIN:ask-docs-auto-generated --> block is untouched
• New <!-- intent-skills:start --> ... <!-- intent-skills:end --> block managed by agents-intent.ts on a byte range strictly disjoint from the ask block
• Idempotent upsert, sibling-preserving, auto-strip when the last entry for a package is removed

Lock schema extension

• NpmLockEntry gains optional format?: 'docs' | 'intent-skills' — default 'docs' so pre-refactor lock entries load unchanged
• ask docs sync adds a second pass that iterates lock entries with format: 'intent-skills' and re-runs the intent adapter
• ask docs remove branches on format: intent entries call removeFromIntentSkillsBlock and drop the lock row

Test plan

• `bun run --cwd packages/schema build` — clean
• `bun run --cwd packages/cli build` — clean
• `bun run --cwd packages/cli lint` — clean
• `bun run --cwd packages/cli test` — 268 pass / 0 fail (237 pre-existing + 31 new)
• SC-3 verified: noise-only fixture (CONTRIBUTING.md + CHANGELOG.md only) fails the quality threshold (quality.test.ts, adapters.test.ts)
• SC-4 verified: all 237 pre-existing add/sync/remove tests pass unchanged
• SC-5 verified: packages/cli builds and lints clean
• Marker isolation verified: seed AGENTS.md with a real ask-docs block, upsert + remove intent entries, ask block preserved byte-for-byte
• SC-1 deferred: live coverage audit against 37 registry entries — scaffold at `packages/cli/scripts/audit-coverage.ts`
• SC-2 deferred: live `tanstack-intent` CLI parity diff — writer matches the canonical snippet in `@tanstack/intent/dist/install-*.mjs`

Deferred follow-ups

• T014 — wire `runRepoDiscovery` through `GithubSource` (requires exposing the extract dir as a post-fetch hook). Adapter + orchestrator are already in place; only the call site remains.
• T025 — end-to-end `ask docs add npm:` integration test that drives the full pipeline against inline fixtures.
• T027 — live coverage audit run (needs all 37 registry packages installed to `node_modules/`).
• T028 — live intent-CLI parity diff (manual: install a real `tanstack-intent` keyword package, run `ask docs add` + `bunx @tanstack/intent install`, diff inside the marker block).

See `Surprises & Discoveries` in the track plan for full rationale.

Commits

• e798a93 — Phase 1: intent-skills format field + @tanstack/intent dep
• 437ce8c — Phase 2: discovery adapter pipeline
• d299919 — Phase 3: dispatch local discovery + sync/remove
• 1f0ff7e — Phase 4: discovery/quality/agents-intent unit coverage
• 398da84 — Phase 5: audit script + CLAUDE.md
• 902042f — self-review: marker isolation test

---

Summary by cubic

Adds a convention-based docs discovery pipeline that runs before the central registry for npm: specs, auto-resolving common docs and wiring @tanstack/intent skills into a dedicated AGENTS.md block. The registry is now a fallback; sync/remove support intent skills. Closes #46.

• New Features

  • Discovery module with adapters: local-ask → local-intent → local-conventions; repo-conventions added. Includes a quality scorer (≥3 markdown files or ≥4 KiB; excludes LICENSE/CHANGELOG/CONTRIBUTING).
  • Intent integration: scans tanstack-intent packages via @tanstack/intent, writes <!-- intent-skills:start --> … <!-- intent-skills:end -->. Idempotent upsert/remove; ask-docs block is untouched.
  • Lock schema: format?: 'docs' | 'intent-skills' (default docs). ask docs sync re-runs intent discovery; ask docs remove cleans the block and lock entry for intent skills.
  • CLI flow: for npm: without --source/--docs-path, run discovery before registry. Docs reuse the existing pipeline; intent skills reference node_modules in place. Adds packages/cli/scripts/audit-coverage.ts for registry coverage.

• Bug Fixes

  • Intent block robustness: end marker search is anchored after its start; replaced the string unescape with a single-pass decoder to preserve quotes and backslashes.
  • Safer scans: validate symlink realpath containment in local-conventions; share MAX_WALK_DEPTH=20 across walkers; meta-file exclusion is now case-insensitive.
  • Sync reliability: runSync refreshes intent-skill blocks even in intent-only projects; the audit script correctly parses YAML alias fields.

^{Written for commit d92ea30. Summary will update on new commits.}

[codecov]

Codecov Report

❌ Patch coverage is 75.48263% with 127 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
packages/cli/src/discovery/repo-conventions.ts	8.57%	64 Missing ⚠️
packages/cli/src/discovery/local-intent.ts	29.72%	52 Missing ⚠️
packages/cli/src/discovery/index.ts	88.00%	3 Missing ⚠️
packages/cli/src/discovery/local-conventions.ts	96.42%	3 Missing ⚠️
packages/cli/src/agents-intent.ts	98.49%	2 Missing ⚠️
packages/cli/src/discovery/quality.ts	96.07%	2 Missing ⚠️
packages/cli/src/discovery/local-ask.ts	97.22%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cloudflare-workers-and-pages]

Deploying ask-registry with    Cloudflare Pages

Latest commit:	d92ea30
Status:	⚡️  Build in progress...

View logs

[cubic-dev-ai]

6 issues found across 21 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/cli/src/discovery/conventions.ts">

<violation number="1" location="packages/cli/src/discovery/conventions.ts:60">
P2: Meta-file exclusion uses case-sensitive exact matches, allowing lowercase/mixed-case CONTRIBUTING/CHANGELOG files to bypass filtering and inflate doc quality scoring.</violation>
</file>

<file name="packages/cli/scripts/audit-coverage.ts">

<violation number="1" location="packages/cli/scripts/audit-coverage.ts:43">
P2: Npm alias parsing only recognizes "- npm:<name>" list items, but registry entries store aliases as objects (ecosystem/name). The regex never matches, leaving npmName undefined and marking all entries as uncovered.</violation>
</file>

<file name="packages/cli/src/discovery/local-conventions.ts">

<violation number="1" location="packages/cli/src/discovery/local-conventions.ts:38">
P1: Validate convention path realpath containment before calling `scoreDirectory`; currently a symlinked docs directory can cause out-of-package recursive traversal during scoring.</violation>
</file>

<file name="packages/cli/src/index.ts">

<violation number="1" location="packages/cli/src/index.ts:789">
P1: `runSync` exits early when `config.docs` is empty, so the newly added intent-skills lockfile resync pass never runs for intent-only projects.</violation>
</file>

<file name="packages/cli/test/discovery/adapters.test.ts">

<violation number="1" location="packages/cli/test/discovery/adapters.test.ts:118">
P3: This assertion block is conditionally skipped for non-`docs` results, so the test can false-pass if discovery returns `intent-skills` unexpectedly.</violation>
</file>

<file name="packages/cli/src/agents-intent.ts">

<violation number="1" location="packages/cli/src/agents-intent.ts:52">
P2: `unescapeDq` performs two sequential unescapes, which drops a literal backslash before a quote (e.g., `foo\"bar` decodes to `foo"bar`). This makes round-tripping existing intent entries lossy when a task or load contains `\"`. Decode with a single-pass escape parser so `\\` and `\"` are handled without double-processing.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant U as User (CLI)
    participant C as CLI Orchestrator
    participant D as Discovery Module
    participant FS as Local File System (node_modules)
    participant R as Central Registry API
    participant AM as AGENTS.md
    participant S as Storage (.ask/docs & .claude/skills)

    Note over U,S: ask docs add npm:<package>

    U->>C: Execute "add" command
    
    rect rgb(240, 240, 240)
        Note over C,D: NEW: Convention-Based Discovery Pipeline
        C->>D: runLocalDiscovery(pkg)
        
        D->>FS: NEW: local-ask (check package.json.ask.docsPath)
        FS-->>D: manifest data
        
        opt No manifest
            D->>FS: NEW: local-intent (check keywords & scan skills)
            FS-->>D: intent skill files
        end
        
        opt No intent skills
            D->>FS: NEW: local-conventions (scan dist/docs, docs, README)
            D->>D: NEW: run quality scoring (>=3 files or >=4KiB)
            FS-->>D: conventional files
        end
        
        D-->>C: DiscoveryResult (kind: 'docs' | 'intent-skills' | null)
    end

    alt Discovery Found Results
        Note over C,S: Discovery Hit Path
        alt kind == 'intent-skills'
            C->>C: NEW: Generate Intent content hash
            C->>AM: NEW: upsertIntentSkillsBlock (disjoint block)
            AM-->>C: success
        else kind == 'docs'
            C->>S: CHANGED: saveDocs (copy from node_modules to .ask/docs)
            C->>S: generateSkill (create .claude/skills/...)
            C->>AM: generateAgentsMd (update ask-docs block)
        end
    else Discovery Miss (null)
        Note over C,R: Fallback Path
        C->>R: resolveFromRegistry(pkg)
        R-->>C: Registry Entry (docsPath/URL)
        C->>S: saveDocs / generateSkill
        C->>AM: generateAgentsMd
    end

    C->>FS: Update ask.lock (NEW: record 'format' field)
    C-->>U: Done! (Success message)

    Note over U,AM: ask docs sync / remove
    U->>C: Execute sync/remove
    C->>FS: Read ask.lock
    alt NEW: format == 'intent-skills'
        C->>D: Re-run localIntentAdapter
        C->>AM: NEW: update/strip intent-skills block
    else format == 'docs'
        C->>S: Standard doc sync/cleanup
    end

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/cli/src/discovery/repo-conventions.ts">

<violation number="1" location="packages/cli/src/discovery/repo-conventions.ts:52">
P2: Quality scoring and file collection use different recursion limits, which can cause valid deep docs roots to be skipped as false negatives.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}