feat(cli): global ASK docs store at ~/.ask/

[amondnet]

Summary

Introduce a global immutable docs store at ~/.ask/ (with ASK_HOME override) so identical <pkg>@<version> entries are fetched once per machine and reused across projects — the same pattern as Cargo, Go modules, pnpm, and Deno.

Closes #55

Changes

Store infrastructure

• packages/cli/src/store/index.ts — resolveAskHome, per-kind path helpers, writeEntryAtomic, acquireEntryLock (with backoff), stampEntry/verifyEntry
• packages/cli/src/store/github-bare.ts — withBareClone for Cargo-style multi-ref efficiency; falls back to null when git is unavailable
• packages/cli/src/store/cache.ts — cacheLs/cacheGc pure functions

Source adapter updates

All four source adapters (npm, github, web, llms-txt) now write into the global store before returning storePath in FetchResult:

• <ASK_HOME>/npm/<pkg>@<version>/
• <ASK_HOME>/github/db/<owner>__<repo>.git/ + <ASK_HOME>/github/checkouts/<owner>__<repo>/<ref>/
• <ASK_HOME>/web/<sha256-of-normalized-url>/
• <ASK_HOME>/llms-txt/<sha256>@<version>/

Materialization modes

New --store-mode flag on ask install / ask add (and ask.json.storeMode):

• copy (default): write full copy into .ask/docs/<pkg>@<v>/ — byte-identical to pre-change behavior
• link: symlink to store, falls back to copy on EPERM/EACCES
• ref: no project-local files; AGENTS.md points at store path directly

Precedence: CLI flag > ask.json.storeMode > 'copy'.

Cache commands

• ask cache ls [--kind npm|github|web|llms-txt] — list entries with sizes
• ask cache gc [--dry-run] — remove entries not referenced by any .ask/resolved.json under \$HOME (or ASK_GC_SCAN_ROOTS)

Schema

• AskJsonSchema.storeMode?: 'copy' | 'link' | 'ref'
• ResolvedEntrySchema.storePath?: string + materialization?: 'copy' | 'link' | 'ref'

Test plan

• `bun run --cwd packages/schema test` — 60 pass
• `bun run --cwd packages/cli test` — 270 pass (34 new tests across 5 new files)
• `bun run --cwd packages/cli lint` — clean
• `bun run --cwd packages/schema build && bun run --cwd packages/cli build` — green
• Manual: `example/` project still installs cleanly with default copy mode (byte-identical output)
• Manual: verify `link` mode on Linux/macOS creates symlinks
• Manual: verify `ref` mode writes AGENTS.md pointing at store path
• Manual: verify `ask cache ls` / `gc` on real store

Backward compatibility

Default `storeMode: copy` keeps output byte-identical at the project level. Existing `ask.json` files parse unchanged. The store is additive — no forced migration.

---

Summary by cubic

Adds a global, immutable docs store at ~/.ask/ (ASK_HOME override) with copy/link/ref modes and cache commands. Speeds installs by reusing <pkg>@<version> across projects and short‑circuiting when a store entry exists. Implements #55.

• New Features

  • Store-hit short-circuit: if a <pkg>@<version> exists in the store, materialize directly without fetching.
  • GitHub bare clone for faster multi-ref fetches, with tar.gz fallback when git is unavailable.
  • Cache commands: ask cache ls and ask cache gc --older-than <duration> (supports s/m/h/d).

• Bug Fixes

  • Atomicity/correctness: added cpDirAtomic for safe directory writes; hashDir uses null-byte separators; URL normalization only lowercases scheme/host.
  • Sources: GitHub extraction now uses cpDirAtomic to avoid partial entries; Web store keys on all sorted URLs to prevent aliasing.
  • CLI/GC: validate --kind before use; cache gc ignores empty scan roots when $HOME is unset; surface readdirSync errors from store reads; only exclude root-level INDEX.md.
  • Security/guardrails: removed shell pipelines in GitHub tar path; strict owner/repo and ref validation; store/write paths are containment-checked. GC only trusts storePath values within ASK_HOME.

^{Written for commit 8830f74. Summary will update on new commits.}

Verification Checklist

Functional:

• Fresh ~/.ask/: bun run --cwd packages/cli build && cd example && node ../packages/cli/dist/index.js install → ~/.ask/npm/next@X/ materialized, .ask/docs/next@X/ copied, bytes match
• Second project: same version → store hit, no network fetch
• --store-mode=link: readlinkSync('.ask/docs/next@X') resolves under ~/.ask/
• --store-mode=ref: AGENTS.md contains absolute <ASK_HOME> path, no .ask/docs/next@X/
• ASK_HOME=/tmp/askh ... → all writes under /tmp/askh/
• ask cache ls lists entries; ask cache gc --dry-run reports; ask cache gc removes
• ask cache gc --older-than 30d only removes entries older than cutoff
• GitHub source with two refs: one <ASK_HOME>/github/db/ bare clone, two checkouts

Non-regression:

• bun run test across all packages green (342 passing)
• bun run --cwd packages/cli lint clean
• Existing example/ project installs and produces byte-identical AGENTS.md vs pre-change (default copy mode)
• ask remove still tears down per-project materialization regardless of store mode

Docs:

• CHANGELOG describes store, ASK_HOME, storeMode, cache ls|gc, github bare-clone
• README "Global Store" section with tree diagram + Windows symlink note

[codecov]

Codecov Report

❌ Patch coverage is 74.60993% with 179 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
packages/cli/src/sources/github.ts	37.93%	54 Missing ⚠️
packages/cli/src/store/index.ts	80.31%	37 Missing ⚠️
packages/cli/src/store/cache.ts	87.16%	24 Missing ⚠️
packages/cli/src/sources/llms-txt.ts	9.09%	20 Missing ⚠️
packages/cli/src/sources/web.ts	9.52%	19 Missing ⚠️
packages/cli/src/sources/npm.ts	67.74%	10 Missing ⚠️
packages/cli/src/store/github-bare.ts	90.54%	7 Missing ⚠️
packages/cli/src/storage.ts	79.31%	6 Missing ⚠️
packages/cli/src/install.ts	96.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cloudflare-workers-and-pages]

Deploying ask-registry with    Cloudflare Pages

Latest commit:	8830f74
Status:	🚫  Build failed.

View logs

[cubic-dev-ai]

0 issues found across 6 files (changes from recent commits).

_{Requires human review: High-impact architectural change introducing a global file store, locking mechanisms, and a garbage collector. Contains documented subtle atomicity risks requiring human verification.}

[cubic-dev-ai]

10 issues found across 30 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="README.md">

<violation number="1" location="README.md:228">
P3: The `storeMode=link` docs are inaccurate: fallback is documented as `EPERM` only, but code falls back on both `EPERM` and `EACCES`.</violation>
</file>

<file name="packages/cli/src/store/cache.ts">

<violation number="1" location="packages/cli/src/store/cache.ts:126">
P1: `cacheGc` defaults to `process.env.HOME` only; when `HOME` is unset, reference scanning is skipped and valid cache entries can be wrongly garbage-collected.</violation>
</file>

<file name="packages/cli/src/sources/web.ts">

<violation number="1" location="packages/cli/src/sources/web.ts:52">
P1: Store keying only on `opts.urls[0]` can alias different multi-URL web configs to the same cache entry, causing incorrect docs to be materialized in `link`/`ref` modes.</violation>
</file>

<file name="packages/cli/src/sources/github.ts">

<violation number="1" location="packages/cli/src/sources/github.ts:134">
P1: Non-atomic store write: if the process crashes between `rmSync` and `cpSync` (or before `stampEntry`), the entry is left corrupt but will pass the `fs.existsSync` store-hit check at the top of `fetch()`, silently serving incomplete docs. Copy to a temp directory first and rename atomically, mirroring the pattern in `writeEntryAtomic`.</violation>
</file>

<file name="packages/cli/CHANGELOG.md">

<violation number="1" location="packages/cli/CHANGELOG.md:11">
P3: The changelog documents `link` mode fallback for `EPERM` only, but this release’s behavior includes `EACCES` fallback too. Update the line to match actual behavior.</violation>
</file>

<file name="packages/cli/src/store/index.ts">

<violation number="1" location="packages/cli/src/store/index.ts:83">
P1: `normalizeUrl` lowercases the entire URL including the path, which is case-sensitive per RFC 3986. This will cause distinct URLs (differing only in path casing) to collide in the store. Only the scheme and host should be lowercased.</violation>

<violation number="2" location="packages/cli/src/store/index.ts:253">
P2: `hashDir` concatenates file names and file contents into the hash without any separator. This creates ambiguity: file `"ab"` with content `"cd"` hashes identically to file `"a"` with content `"bcd"`. Add a null-byte or length prefix between fields to ensure unique hash inputs.</violation>
</file>

<file name="packages/cli/src/index.ts">

<violation number="1" location="packages/cli/src/index.ts:264">
P2: Validate `--kind` before passing it to `cacheLs`; the current cast allows invalid values and can make `cache ls` traverse paths outside `ASK_HOME`.</violation>
</file>

<file name="packages/cli/src/install.ts">

<violation number="1" location="packages/cli/src/install.ts:469">
P1: Do not silently skip unreadable store files; it can produce partial installs while reporting success.</violation>

<violation number="2" location="packages/cli/src/install.ts:474">
P2: `readFilesFromStore` drops all `INDEX.md` files, including nested real docs. Only the root generated index should be excluded.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant User as User/CLI
    participant Runner as Install Runner
    participant Adapter as Source Adapter (npm/github/web)
    participant Store as Global Store Logic (~/.ask/)
    participant GFS as Global FileSystem
    participant LFS as Local Project (.ask/)
    participant Remote as External API/Git

    Note over User,Remote: NEW: Global Store Installation Flow

    User->>Runner: ask install / add
    Runner->>Store: resolveAskHome() (ASK_HOME or ~/.ask/)
    
    rect rgb(240, 240, 240)
        Note right of Runner: Store Hit Check
        Runner->>GFS: NEW: Check if entry exists for <pkg>@<version>
        alt NEW: Entry Found (Store Hit)
            GFS-->>Runner: Existing Path
        else NEW: Entry Missing (Store Miss)
            Runner->>Adapter: fetch(spec)
            
            alt NEW: GitHub Source (Optimized)
                Adapter->>Store: withBareClone()
                Store->>GFS: Check shared bare repo in /github/db/
                alt Bare Repo Missing
                    Store->>Remote: git clone --bare
                else Bare Repo Exists
                    Store->>Remote: git fetch origin <ref>
                end
                Store->>GFS: git archive to /github/checkouts/
            else npm/Web/llms-txt
                Adapter->>Remote: Download tarball/content
            end

            Adapter->>Store: NEW: acquireEntryLock(entryPath)
            Store->>GFS: Create <entry>.lock (with backoff)
            
            Adapter->>Store: NEW: writeEntryAtomic()
            Store->>GFS: Write to .tmp dir -> Atomic Rename
            
            Adapter->>Store: NEW: stampEntry()
            Store->>GFS: Write .ask-hash (integrity)
            
            Adapter->>Store: release lock
            Store->>GFS: unlink <entry>.lock
            
            Adapter-->>Runner: FetchResult (includes storePath)
        end
    end

    rect rgb(230, 245, 255)
        Note right of Runner: Materialization Mode (CLI flag or ask.json)
        alt NEW: storeMode == 'link'
            Runner->>LFS: CHANGED: symlink .ask/docs/ -> ~/.ask/entry
            Note over LFS: Fallback to copy on EPERM
        else NEW: storeMode == 'ref'
            Runner->>Runner: Skip local doc creation
        else storeMode == 'copy' (Default)
            Runner->>LFS: Recursive copy from Global Store
        end
    end

    Runner->>LFS: CHANGED: Update resolved.json (storePath + materialization)
    Runner->>LFS: CHANGED: generateAgentsMd() (point to Local or Global path)
    Runner-->>User: Done

    Note over User,GFS: Cache Management (Separate Flow)
    User->>Runner: ask cache ls
    Runner->>Store: cacheLs()
    Store->>GFS: Walk kind directories (npm, github, etc)
    Runner-->>User: Display entries and sizes

    User->>Runner: ask cache gc
    Runner->>LFS: Scan roots for .ask/resolved.json
    Runner->>Store: cacheGc(referencedPaths)
    Store->>GFS: NEW: Remove unreferenced entries (olderThan filter)

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

0 issues found across 8 files (changes from recent commits).

_{Requires human review: Large architectural change introducing a global store, new CLI commands, and complex materialization logic (linking/copying) with known atomicity nuances requiring human review.}