Skip to content

feat: crate package support#4

Merged
harlan-zw merged 1 commit intoskilld-dev:mainfrom
oritwoen:feat/crate-sync-namespace
Apr 22, 2026
Merged

feat: crate package support#4
harlan-zw merged 1 commit intoskilld-dev:mainfrom
oritwoen:feat/crate-sync-namespace

Conversation

@oritwoen
Copy link
Copy Markdown
Collaborator

Description

This PR adds Rust crate support to skilld with crate:<name> (e.g. skilld add crate:serde), while preserving existing npm/git behavior.

What’s included:

  • crates.io resolver implementation (src/sources/crates.ts)
  • crate-aware sync routing (src/commands/sync.ts)
  • mixed-batch support without degrading npm parallel sync
  • npm/crate collision prevention via crate storage namespace (@skilld-crate/<name>) and crate identity (crate:<name>) in lock metadata
  • shared fetch/rate-limit helper improvements in src/sources/utils.ts
  • README updates for new usage examples

Tests:

  • test/unit/sources-crates.test.ts
  • test/unit/sync-crate-routing.test.ts
  • test/unit/sync-crate-version.test.ts
  • test/e2e/crate-smoke.test.ts

Validation:

  • unit tests for crate paths pass
  • crate e2e smoke test passes
  • pnpm typecheck passes
  • pnpm build passes

Linked Issues

N/A

Additional context

Please focus review on namespace separation and cache/identity consistency in src/commands/sync.ts.

@harlan-zw
Copy link
Copy Markdown
Collaborator

This is a really cool direction. I'll need to get back to you on it in the next couple of days.

@oritwoen
Copy link
Copy Markdown
Collaborator Author

Sure, I tried to make it as agnostic as possible to open up support for other packages like Go, Python, etc. in next pull requests, but I feel like I'm missing something. Anyway, thanks for taking a look when you have a moment :)

@harlan-zw
Copy link
Copy Markdown
Collaborator

harlan-zw commented Feb 25, 2026

So, thinking a bit on this, I think it definitely has its place, but there are a few items on the roadmap I want to sort before we ship cross-language support.

  • v1.0 - Official release (~ next monday)
  • v1.1 - Maintainer mode (allow maintainers to self-publish their own skills as part of releases) (~next week sometime)
  • v1.2 - ?
  • v2 - Rust / Python / etc support

My current concern is that going too broad before there's enough feedback will make quicker iterations more difficult. But I'm keen to ship this stuff as quickly as possible as the package needs to gain as much momentum as it can while still nailing the DX so I don't think we need to wait too long to merge (2-3 weeks).

@oritwoen
Copy link
Copy Markdown
Collaborator Author

oritwoen commented Feb 25, 2026

I was also thinking about this after submitting this pull request and will change it to a draft for now.

I was wondering what exactly was missing from an agnostic perspective. I've been using Rust and Python, and various languages ​​for agents depending on my needs, which is where this PR came from. However, after its creation, I realized there was a gap in the node ecosystem itself. There was no package that aggregated information from various registries from different languages ​​into a single form, cached it, and made it usable offline.

I'm currently working on:

Universal package registry client. Query npm, PyPI, crates.io, RubyGems, Packagist, and more with a single PURL-native API.

And once that's released, I'll modify this PR to support my package, which will then support all registries simultaneously.

And thanks so much for skilld!

@oritwoen oritwoen marked this pull request as draft February 25, 2026 15:55
@harlan-zw
Copy link
Copy Markdown
Collaborator

That sounds really useful, I would be very happy to receive that PR :)

@harlan-zw harlan-zw marked this pull request as ready for review April 22, 2026 13:53
Adds `skilld add crate:<name>` for generating skills from crates.io docs.
Routed through the new prefix parser alongside npm/gh/@-prefixes.

- SkillSource gains a `crate` variant; `resolveSkillName` returns `crate:<name>`
- New resolveCrateDocsWithAttempts hits crates.io, prefers docs.rs for
  versioned URLs, enriches from GitHub when applicable, and discovers llms.txt
- Rate-limited via shared createRateLimitedRunner; SKILLD_USER_AGENT constant
  and retry-on-429/5xx for ofetch consumers
- Storage/identity split: lockfile stores crate:<name>, cache/symlinks use
  @skilld-crate/<name> to avoid colliding with npm packages of the same name
- Single-package path serializes crate syncs (respects crates.io rate limit)
  while npm batches keep the parallel path
- toStoragePackageName applied to prepare/install/status/search-helpers so
  restore, doc counts, and search index lookups resolve crate cache paths
- Bulk `skilld update` now includes all installed crate skills
@harlan-zw
Copy link
Copy Markdown
Collaborator

harlan-zw commented Apr 22, 2026

Force-pushed a rewrite onto this branch that rebuilds crate support on top of the new prefix parser (src/core/prefix.ts) landed in #59, replacing the ad-hoc isCrateSpec checks in sync.ts.

What changed from the original PR

  • SkillSource gains a crate variant; parseSkillInput('crate:serde@1.0') returns typed { type: 'crate', package, version }
  • addCommandDef routes crate entries through the existing classification switch alongside npm / git / @curator
  • Shared toStoragePackageName helper in src/core/prefix.ts maps crate:<name>@skilld-crate/<name>; applied in sync, prepare, install, status, and search-helpers so restore / doc counts / search index lookups all resolve crate cache paths (missed in the original)
  • Bulk skilld update now re-resolves all installed crate skills (they never appear in state.outdated since they're not in package.json)
  • enhanceSkillWithLLM gains a cachePackageName option so LLM section cache uses the namespaced key while display stays human-friendly

What was kept from the original

  • resolveCrateDocsWithAttempts in src/sources/crates.ts (crates.io → docs.rs → GitHub enrichment → llms.txt discovery)
  • Rate-limited createRateLimitedRunner and SKILLD_USER_AGENT in src/sources/utils.ts
  • All four test files (sources-crates, sync-crate-routing, sync-crate-version, crate-smoke)

Verification

  • pnpm typecheck passes
  • 793/793 unit tests pass (40 new crate tests)
  • pnpm build emits resolveCrateDocsWithAttempts

@harlan-zw harlan-zw changed the title feat: add crate package sync with isolated storage namespace feat: crate package support Apr 22, 2026
@harlan-zw harlan-zw merged commit 111d97d into skilld-dev:main Apr 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants