Skip to content

fix(cli): stop downloading full GitHub monorepo for scoped npm specs#39

Merged
amondnet merged 1 commit into
mainfrom
fix/scoped-npm-registry-lookup
Apr 9, 2026
Merged

fix(cli): stop downloading full GitHub monorepo for scoped npm specs#39
amondnet merged 1 commit into
mainfrom
fix/scoped-npm-registry-lookup

Conversation

@amondnet

@amondnet amondnet commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Summary

ask docs add npm:@mastra/client-js was extremely slow because three bugs stacked up and ultimately caused the CLI to download the entire mastra-ai/mastra monorepo tarball instead of the single scoped package. This PR fixes all three.

Root cause

  1. parseEcosystem bailed on scoped names. The old guard (!input.includes('/')) made npm:@mastra/client-js return ecosystem: undefined, so the registry lookup ran with a garbage spec (registry/npm/npm:).
  2. fetchRegistryEntry did not URL-encode segments. Even with a correct name, @mastra/client-js would have collided with the server's two-segment slug parser.
  3. Registry miss fallback was hard-coded to source: 'github'. The npm ecosystem resolver extracts repository.url from npm metadata and hands off to GithubSource — which downloaded the upstream repo in full.

The user's npm: prefix was an explicit request to use npm, so the fallback ignored their intent.

Fixes

  • packages/cli/src/registry.ts
    • parseEcosystem now splits on the first colon as long as it appears before any slash. Scoped ecosystem specs parse correctly; owner/repo shorthand still bypasses the branch.
    • fetchRegistryEntry URL-encodes both path segments so scoped names survive the round trip.
  • apps/registry/server/api/registry/[...slug].get.ts — decodes each catch-all segment. Nitro only auto-decodes %40, so %2F has to be handled explicitly.
  • packages/cli/src/index.ts — on a registry miss with an explicit npm: prefix, the CLI now fetches via NpmSource (single tarball + node_modules local-first) instead of the github resolver fallback. The library name is slugified the same way the registry server does (@mastra/client-jsmastra-client-js) and the original package name is preserved on the package field.
  • packages/cli/test/registry.test.ts — 4 regression tests covering parseEcosystem for scoped names, simple prefixes, and owner/repo shorthand.

Test plan

  • bun run --cwd packages/cli test — 259 pass (was 255)
  • bun run --cwd packages/cli lint — clean
  • Manual end-to-end: ask docs add npm:@mastra/client-js in an empty project now fetches 25 doc files from the single npm tarball in seconds and writes .ask/docs/mastra-client-js@1.13.2/. Previously it would try to download the full mastra-ai/mastra monorepo.

Notes

  • CLI + server changes are bundled because the URL-encoding fix requires both sides to agree on the round trip.
  • Pre-existing Nitro queryCollection type diagnostics in [...slug].get.ts are unrelated to this change.

Summary by cubic

Fixes CLI handling of scoped npm specs so we fetch a single npm tarball instead of the full GitHub monorepo. ask docs add npm:@mastra/client-js is now fast and writes docs for the correct package.

  • Bug Fixes

    • Parse ecosystem prefix only when the colon comes before any slash; scoped specs like npm:@mastra/client-js parse correctly.
    • URL-encode both registry path segments so scoped names don’t get split server-side.
    • Decode %2F in server slug segments to round-trip scoped names reliably.
    • On registry miss with an explicit npm: prefix, fetch via NpmSource, slugify the directory name (@mastra/client-jsmastra-client-js), and keep the original package field.
  • Tests

    • Added 4 regression tests for parseEcosystem and owner/repo shorthand.
    • CLI test suite now at 259 passing.

Written for commit 3d3aac1. Summary will update on new commits.

`ask docs add npm:@mastra/client-js` was extremely slow because three
bugs stacked up:

1. `parseEcosystem` bailed on any input containing `/`, so scoped names
   like `npm:@mastra/client-js` were returned with `ecosystem: undefined`
   and a garbage spec, producing a `registry/npm/npm:` lookup URL.
2. `fetchRegistryEntry` did not URL-encode segments, so even a correct
   scoped name would have collided with the server's two-segment slug
   parser.
3. On a registry miss with `npm:` prefix, the CLI fell back to the npm
   ecosystem resolver, which extracts the repository URL from npm
   metadata and hard-codes `source: 'github'`. For `@mastra/client-js`
   that meant downloading the entire `mastra-ai/mastra` monorepo
   tarball — exactly the surprise the user hit.

Fixes:
- `parseEcosystem` now splits on the first colon as long as it appears
  before any slash, so scoped ecosystem specs parse correctly while
  `owner/repo` shorthand still bypasses the branch.
- `fetchRegistryEntry` encodes both path segments so scoped names
  survive the round trip to the registry API.
- The registry server decodes each catch-all segment, since Nitro only
  decodes `%40` automatically and leaves `%2F` intact.
- On a registry miss with an explicit `npm:` prefix, honor the user's
  intent and fetch the single npm tarball via `NpmSource` instead of
  pulling the upstream repo. The library name is slugified the same way
  the registry server slugifies monorepo entries (`@mastra/client-js`
  → `mastra-client-js`) and the original package name is preserved on
  the `package` field so the tarball fetch and `node_modules` local
  read both resolve correctly.

Verified end-to-end against `/tmp/ask-test`: registry lookup now logs
the correct spec, the npm tarball path fetches 25 doc files in seconds,
and the slugified directory (`mastra-client-js@1.13.2`) is written
without the monorepo detour.
@codecov

codecov Bot commented Apr 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying ask-registry with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3d3aac1
Status: ✅  Deploy successful!
Preview URL: https://104d3beb.ask-registry.pages.dev
Branch Preview URL: https://fix-scoped-npm-registry-look.ask-registry.pages.dev

View logs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Requires human review: This PR modifies both CLI source resolution logic and a server-side API route handler, which requires architectural context to ensure no regressions in package fetching.

Architecture diagram
sequenceDiagram
    participant User
    participant CLI as CLI (packages/cli)
    participant Registry as Registry API (Nitro Server)
    participant NPM as NPM Registry
    participant GH as GitHub API/Tarball

    User->>CLI: ask docs add npm:@mastra/client-js
    
    CLI->>CLI: CHANGED: parseEcosystem()<br/>(Identifies 'npm' before the scoped '/')
    
    CLI->>Registry: CHANGED: GET /api/registry/npm/%40mastra%2Fclient-js<br/>(URL-encodes scoped name)
    
    Registry->>Registry: NEW: Decode slug segments<br/>(Handles %2F for scoped names)
    
    alt Registry Hit
        Registry-->>CLI: Return package metadata
    else Registry Miss (404)
        Registry-->>CLI: Not Found
        
        CLI->>CLI: NEW: Check ecosystem == 'npm'
        
        alt Ecosystem is 'npm' (NEW Path)
            CLI->>CLI: NEW: Slugify dir name (@mastra/client-js -> mastra-client-js)
            CLI->>NPM: Fetch specific package tarball
            NPM-->>CLI: Return tarball data
        else Other Ecosystem (Fallback)
            CLI->>GH: Download full repository source
            GH-->>CLI: Return large monorepo tarball
        end
    end

    CLI->>CLI: Extract docs to .ask/docs/<name>
    CLI-->>User: Success
Loading

@amondnet amondnet merged commit 0dc5ce2 into main Apr 9, 2026
5 checks passed
@amondnet amondnet deleted the fix/scoped-npm-registry-lookup branch April 9, 2026 01:42
This was referenced Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant