RFC-0005 and RFC-0006: Skill Registry and Harness Integration#10
RFC-0005 and RFC-0006: Skill Registry and Harness Integration#10jwm4 wants to merge 11 commits intomlflow:mainfrom
Conversation
Propose a governed, metadata-first registry for AI agent skills in MLflow with typed source pointers, lifecycle management, security scan tracking, skill groups, and federated discovery. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds RFC-0005 proposing a new governed, metadata-first Skill Registry for MLflow, including versioning, lifecycle/publish-state governance, typed source pointers (Git/OCI/ZIP), and first-class skill groups.
Changes:
- Introduces the Skill Registry conceptual model (skills, skill versions, tags, aliases) and publish-state lifecycle.
- Defines skill groups with versioned membership snapshots, plus associated tags/aliases.
- Specifies proposed DB schema (12 tables) and API surfaces (REST, Python SDK, CLI, UI).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| published skills, not the process of writing them. | ||
| - **Skill format specification.** The registry is format-agnostic. It | ||
| does not define or enforce what a skill looks like (SKILL.md, plugin | ||
| manifests, etc.). |
There was a problem hiding this comment.
Could we consider MLflow artifact storage as a first-class source type here? That would give us a natural UI upload flow and keeps the door open for more MLflow-native packaging/optimization later, even if Git and OCI stay the main paths.
There was a problem hiding this comment.
Good idea. We've added mlflow as an explicit future source type in the "Source type extensibility" section — it would allow storing skill content directly in MLflow artifact storage, providing a natural UI upload flow. It's deferred from the initial implementation to keep the registry metadata-first, but can be added as a follow-up without breaking changes. The adoption strategy's "Follow-up" section also calls this out.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| RETIRED = "retired" | ||
|
|
||
|
|
||
| class SkillSourceType(StrEnum): |
There was a problem hiding this comment.
Can we line this lifecycle up more closely with the MCP registry RFC in PR 12 (#12)? Right now this introduces draft, published, deprecated, and retired plus a separate skill-level status, which makes the cross-registry pattern feel less consistent unless there’s a strong skill-specific reason to diverge.
There was a problem hiding this comment.
Fully aligned — replaced the 4-state publish lifecycle (draft/published/deprecated/retired) with RFC-0004's 3-state model using status (active/deprecated/deleted). SkillStatus is now the only lifecycle enum. Parent entity status (Skill.status, SkillGroup.status) is derived from the latest version rather than set independently.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| does not perform scans. Scanning tools are separate. | ||
| - **Agent harness integration.** How a specific agent harness (Claude | ||
| Code, Codex, Cursor, etc.) installs or loads skills from the registry | ||
| is outside this RFC. The registry provides the metadata; harness |
There was a problem hiding this comment.
Do we want one short paragraph on external skill conventions we expect to interoperate with? Even if the registry stays format-agnostic, it would help to say whether we’re aligning with existing Claude/Cursor/GitHub-style skill packaging or deliberately avoiding standardization for now.
There was a problem hiding this comment.
Done — fully aligned with RFC-0004. Version lifecycle now uses status with three states (active, deprecated, deleted), matching MCPStatus. New versions default to active. Skill.status and SkillGroup.status are now read-only, derived from the latest version's status. Also added latest_version_alias on both Skill and SkillGroup, following RFC-0004's pattern for deterministic "latest" resolution. Store interface switched from AbstractSkillRegistryStore with @abstractmethod to SkillRegistryMixin with raise NotImplementedError, matching MCPServerRegistryMixin.
(Comment from Claude Code under the supervision of Bill Murdock.)
| #### SkillVersion | ||
|
|
||
| A versioned record containing a typed source pointer, publish state, | ||
| and tags. |
There was a problem hiding this comment.
I initially expected aliases to live on SkillVersion, but after checking MLflow model registry, the existing pattern is that aliases are owned by the top-level entity and point to a version. If the goal is consistency with MLflow and PR 12 (#12), I think this design is fine. It may just be worth making that pattern more explicit in the text.
There was a problem hiding this comment.
Aligned — set_skill_alias takes (name, alias, version) and sets the alias on the Skill entity, pointing to a specific version. Same pattern for set_skill_group_alias. This matches the MCP RFC's approach where aliases are owned by the parent entity rather than the version.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| ) | ||
| ``` | ||
|
|
||
| ## Create a skill group with a versioned membership snapshot |
There was a problem hiding this comment.
What's the justification for skill groups? Why not just using tags on skills
There was a problem hiding this comment.
Added a "Why groups instead of tags?" section to the RFC. The key reasons: (1) groups pin specific member versions for reproducibility, (2) groups have their own lifecycle and aliases independent of member status, (3) groups can reference capabilities across registries (skill registry + MCP registry), (4) groups map naturally to the "plugin" concept in agent harnesses, and (5) tags are flat key-value pairs that can't express ordered, versioned membership.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| trace data back to a governed skill record to understand adoption | ||
| across an organization. | ||
|
|
||
| ### Use cases |
There was a problem hiding this comment.
Could we add explicit JTBDs per persona (developer, admin etc.)? It would make it easier to evaluate which parts of the data model are required
There was a problem hiding this comment.
Done — replaced the abstract use case bullets with three end-to-end persona flows: a platform administrator (registers, scans, groups, aliases, deprecates), a developer (searches, resolves alias, pulls), and a security engineer (queries scan tags, deprecates failed versions, tracks compliance). These ground the data model in concrete user workflows.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
- Switch to YAML frontmatter to match repo template - Make ER diagram a mermaid code block - Rename source_url → source (MLflow consistency) - Rename content_hash → content_digest (OCI alignment) - Use SkillSourceType enum consistently for alias/membership types - Fix Skill.aliases type to list[SkillAlias] (was dict, missing source_type) - Fix search_skills example to use search_skill_versions (publish_state is on version) - Clarify alias resolve returns both version and source_type - Specify valid filter fields per search endpoint - Fix delete semantics: skill delete blocked when versions referenced by groups - Drop Databricks dogfooding anecdote - Remove "Relationship to other AI asset registries" section - Remove "Impact on existing MLflow components" table - Condense SDK/CLI sections (defer to store interface + examples) - Tighten drawbacks and trim weaker alternatives Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove redundant group_name/group_version/workspace fields from the entity — parent identity is provided by the enclosing SkillGroupVersion. The DB schema retains those columns as FKs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update: Review Feedback Round 1Pushed changes addressing the straightforward review feedback: YAML frontmatter, mermaid ER diagram, field renames ( Still openStructural:
Scope:
Content:
(Comment from Claude Code under the supervision of Bill Murdock.) |
Version uniqueness is now (name, version) instead of (name, version, source_type). source_type and source are optional fields on SkillVersion. Cascaded this change through entities, DB schema, store interface, REST API paths, examples, and CLI. Added RFC-0006 covering harness-specific installation with adapters for Claude Code, Codex CLI, Cursor, and Antigravity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-27)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Skill Registry Harness Integration) — new:
Resolved review comments (newly replied)
Still open
(Comment from Claude Code under the supervision of Bill Murdock.) |
Move marketplace.json from Alternatives into Detailed Design with full endpoint spec, response format, configuration, and limitations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jwm4
changed the title
|
Also, I've converted this to draft given all the open issues listed above. |
- Lifecycle: publish_state → status, 3 states (active/deprecated/ deleted), derived parent status. Aligns with RFC-0004. - Add latest_version_alias to Skill and SkillGroup. - Store: AbstractSkillRegistryStore → SkillRegistryMixin with NotImplementedError. Add order_by to search methods. - Cross-registry membership: rename skill_name/skill_version to member_name/member_version, add registry field (skill/mcp). - Conditional FK for MCP refs → application-layer enforcement. - Pull clarified as client-side, removed from store mixin and REST API. - Dual MCP registration: MCP registry is default, kind=mcp-server reserved for embedded configs only. - SDK namespace: mlflow.skills → mlflow.genai.skills. - Add external skill conventions paragraph. - Add skill groups justification (why not just tags). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document `mlflow` as a deferred source type in the extensibility section and adoption strategy follow-up, per review feedback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-29)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Harness Integration):
Still open (deferred to future pass)
(Comment from Claude Code under the supervision of Bill Murdock.) |
- Replace abstract use case bullets with end-to-end persona flows
(platform admin, developer, security engineer)
- Expand security scan tracking with structured tag namespace convention
(scan.{type}.{field}) and documented fields/examples
- Add permissions section mapping operations to MLflow's READ/EDIT/MANAGE
levels, with status transitions and alias management requiring MANAGE
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-29, part 2)The four items listed as "still open" in the previous update are now all addressed:
All review comments from mprahl, serena-ruan, and Copilot now have replies. I don't have further edits planned at this time. The RFC is ready for another round of review. (Comment from Claude Code under the supervision of Bill Murdock.) |
| rfc_pr: https://github.com/mlflow/rfcs/pull/10 | ||
| --- | ||
|
|
||
| # RFC: Skill Registry |
There was a problem hiding this comment.
@Bill Murdock — Reading through the RFC and the proposed API, one thing struck me: given that the registry already supports kind=[skill, agent, mcp-server, hook], should we consider a more generic name than "skills" / "skill groups"?
The data model is already kind-agnostic — a Skill entity with kind=agent is conceptually an asset that happens to be an agent, not a skill. Calling the registry an Asset Registry (or Capability Registry) and using create_asset() / create_asset_group() instead of create_skill() / create_skill_group() would better reflect what the API actually does and avoid the naming tension where "skill" is both a specific kind and the umbrella term for all kinds.
The REST paths and CLI would follow naturally: mlflow assets create, mlflow asset-groups create-version, etc.
| Add a Skill Registry to MLflow: a governed, metadata-first registry for | ||
| AI agent capabilities. The registry stores metadata and typed source | ||
| pointers (to Git repos, OCI registries, ZIP archives, etc.) rather | ||
| than artifacts directly. It provides enterprise governance on top of |
There was a problem hiding this comment.
Could we support both approaches? The user would either set pointers to external storage (Git, OCI, ZIP) or store assets directly in MLflow's internal artifact store.
The reason: source pointers are the right default for customers who already manage assets in Git or other systems. But for customers who don't have that infrastructure in place, or who already use MLflow to store models and want their agent capabilities alongside them, or who operate in airgapped or restricted environments where reaching external sources isn't practical — the ability to store assets directly in the registry would be a better fit.
There was a problem hiding this comment.
Agreed. The latest revision adds source_type="mlflow" as a first-class source type in the initial implementation. Content is stored as a directory tree of individual files in MLflow's artifact store, consistent with how model artifacts are stored. Users can choose external source pointers (Git, OCI, ZIP) or direct artifact storage depending on their environment. This covers the airgapped and "capabilities alongside models" use cases you described.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
|
|
||
| ### Out of scope | ||
|
|
||
| - **Artifact storage.** The registry stores metadata and source |
There was a problem hiding this comment.
given that mlflow already has an "artifact store" to store models and traces, I think the proposal could just embrace it and support it as an option, as per my previous comment.
There was a problem hiding this comment.
Addressed in the same change. See reply to your earlier comment.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
| permissions. | ||
| - **Alias management requires MANAGE.** Aliases like `production` | ||
| control which version downstream consumers resolve to. Changing an | ||
| alias has the same blast radius as a status transition. |
There was a problem hiding this comment.
Given the impact and blast radius of alias management, should alias changes be auditable?
Currently, last_updated_by / last_updated_timestamp on the parent Skill captures that something changed, but doesn't record alias-specific history (which alias, what it pointed to before, who changed it, when). For enterprise governance scenarios — "who promoted this to production and when?" or "what was production pointing to yesterday before the incident?" — an alias change log would be valuable.
This could be as lightweight as an append-only skill_alias_history table (alias, old_version, new_version, changed_by, timestamp), or even just emitting an event that external audit systems can consume. Worth considering for the initial design, since retrofitting audit onto alias mutations is harder than including it upfront.
There was a problem hiding this comment.
Added. Every alias mutation (set or delete) now appends a record to an append-only history table with the alias name, old version, new version, who changed it, and when. This covers "who promoted this to production?" and "what was production pointing to before the incident?" queries.
The design includes:
SkillAliasHistoryandSkillGroupAliasHistoryentitiesskill_alias_historyandskill_group_alias_historydatabase tables (append-only, no updates or deletes through the API)get_skill_alias_history()andget_skill_group_alias_history()store methods- REST endpoints at
/{name}/aliases/historyand/{name}/aliases/{alias}/history
History is recorded automatically by the store on every alias mutation. No separate API call needed.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
| cross-registry concern that applies equally to skills, MCP servers, | ||
| and other AI asset registries. It is expected to be solved at the | ||
| platform level across all MLflow registries rather than piecemeal in | ||
| each one. |
There was a problem hiding this comment.
Cross-workspace visibility can be managed at the infrastructure level (e.g., Kubernetes RBAC in OpenShift AI), so deferring that is reasonable. However, the RFC should consider supporting export/import of assets between workspaces — or at minimum, a move/duplicate operation. Use case: a platform team develops and tests assets in a staging workspace or even on a staging instance, then promotes them to a production workspace or instance. Without export/import, they'd need to re-register everything manually, including versions, tags, aliases, and group memberships. Even a simple mlflow skills export --name code-review --output bundle.json / mlflow skills import --input bundle.json --workspace production would cover this.
There was a problem hiding this comment.
Good use case. We'd like to follow the other MLflow registries on this. Neither the model registry nor the MCP registry (RFC-0004) has export/import yet, and we want to stay consistent rather than designing a serialization format in isolation. If export/import lands as a cross-registry capability in MLflow, the skill registry should adopt the same approach.
Added a mention in the adoption strategy's follow-up section. The data model is fully serializable, so this can be built on top of the existing API without schema changes.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
- Promote source_type="mlflow" from deferred to first-class: directory tree storage in MLflow artifact store, consistent with model artifacts - Add alias audit trail: append-only history tables, entity definitions, store methods, and REST endpoints for both skills and skill groups - Add cross-workspace export/import as explicit follow-up item - Update RFC-0006 to use source-agnostic language Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-30)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Harness Integration):
Open discussion
All review comments from mprahl, serena-ruan, Copilot, and etirelli now have replies (except the naming question, which is under active discussion). (Comment from Claude Code under the supervision of Bill Murdock.) |
| class SkillStatus(StrEnum): | ||
| ACTIVE = "active" | ||
| DEPRECATED = "deprecated" | ||
| DELETED = "deleted" |
There was a problem hiding this comment.
We added back a fourth DRAFT status in the mcp registry proposal fyi. If feedback on it is positive, might be useful here too?
bef24e7
| schema changes since the column stores a string value. `kind` is | ||
| immutable after creation. | ||
|
|
||
| **MCP servers: two registration paths.** The MCP server registry |
There was a problem hiding this comment.
Do we need both paths? I can see the benefit when embedded as part of an artifact, but wanted to double check if we can provide a cleaner path like a way for an embedded mcp to be automatically registered. Sorry if this was already discussed!
|
|
||
| ```python | ||
| @dataclass | ||
| class SkillGroup: |
There was a problem hiding this comment.
Could it make sense for skill group to simply be a Skill with kind=group? A skill can already be pointing to any number of things. Could potentially simplify/unify the data model a bit. Just putting it out there as an idea!
Summary
Related Issues
Numbering
This RFC is numbered 0005, assuming the MCP Server Registry proposal (mlflow/mlflow#22625) will be RFC-0004.
Design Highlights
mlflow.skills), and CLI (mlflow skills,mlflow skill-groups)🤖 Generated with Claude Code
This comment comes from Claude Code under the supervision of Bill Murdock.