feat(registry): [FEATURE] Add package registry support for dependency…#1471
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Introduces an experimental, REST-based “registries” package source alongside the existing git-based resolver, including lockfile v2 support, install/policy wiring, publishing, and documentation.
Changes:
- Add dedicated registry resolver/client/auth/extraction utilities and wire them into install + drift detection + lockfile semantics.
- Add
registries:support inapm.yml, registry-aware drift/policy enforcement (registry_source), and a newapm publishcommand. - Add extensive unit/e2e tests and documentation for registry workflows, governance, and security model.
Reviewed changes
Copilot reviewed 66 out of 67 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_lockfile_v2_bump.py | New tests for opportunistic lockfile v1→v2 bump + round-trip fields |
| tests/unit/test_install_update.py | Mock updated for source semantics |
| tests/unit/test_install_command.py | Make output assertion robust to rich wrapping |
| tests/unit/test_drift_registry.py | New registry-aware drift + download-ref tests |
| tests/unit/test_apm_yml_registries.py | New tests for registries: parsing + default routing |
| tests/unit/registry/test_semver.py | New tests for semver helpers used by registry resolver |
| tests/unit/registry/test_resolver_e2e_http.py | New e2e registry resolver tests with real local HTTP server |
| tests/unit/registry/test_resolver.py | New resolver orchestration tests with fake client |
| tests/unit/registry/test_extractor.py | New extraction + hash/traversal security tests |
| tests/unit/registry/test_auth.py | New registry auth/env-var lookup tests |
| tests/unit/policy/test_run_dependency_policy_checks.py | Add tests for registry source policy wiring + transitive enforcement |
| tests/unit/marketplace/test_registry_integration.py | New marketplace.json registry routing tests |
| tests/unit/install/test_resolve_registry.py | New tests for resolve-phase registry resolver wiring + lockfile replay |
| tests/unit/install/test_policy_gate_phase.py | Ensure registries + transitive deps plumbed into policy gate |
| tests/unit/core/test_experimental.py | Register new experimental flags + invariants tests |
| src/apm_cli/policy/schema.py | Add RegistrySourcePolicy to policy schema |
| src/apm_cli/policy/policy_checks.py | Implement _check_registry_source + plumb registries kwarg |
| src/apm_cli/policy/parser.py | Parse registry_source policy block |
| src/apm_cli/policy/install_preflight.py | Plumb registries map into policy preflight runner |
| src/apm_cli/policy/init.py | Export RegistrySourcePolicy |
| src/apm_cli/models/dependency/reference.py | Add source/registry_name + registry object-form parsing |
| src/apm_cli/models/apm_package.py | Parse registries: block + default routing pass |
| src/apm_cli/marketplace/models.py | Add marketplace plugin registry field + parsing/validation |
| src/apm_cli/install/sources.py | Route registry downloads via registry resolver; record resolution into lockfile |
| src/apm_cli/install/registry_wiring.py | New helper module to keep registry orchestration out of sources.py |
| src/apm_cli/install/phases/resolve.py | Construct registry resolver and implement registry download callback + lock replay |
| src/apm_cli/install/phases/policy_target_check.py | Plumb registries map into dependency policy checks |
| src/apm_cli/install/phases/policy_gate.py | Forward manifest registries into policy checks |
| src/apm_cli/install/phases/integrate.py | Skip git probing for non-git sources; cached skip logic for registry deps |
| src/apm_cli/install/context.py | Add registry_resolver to install context |
| src/apm_cli/drift.py | Add registry-aware drift logic and lockfile replay ref overrides |
| src/apm_cli/deps/registry_proxy.py | Docs clarify distinction between proxy registry vs dedicated registries |
| src/apm_cli/deps/registry/semver.py | New semver helpers for registry routing/resolution |
| src/apm_cli/deps/registry/feature_gate.py | New experimental feature gate for registries |
| src/apm_cli/deps/registry/extractor.py | New archive verification + safe extraction for registry downloads |
| src/apm_cli/deps/registry/config_loader.py | New merged registry config precedence loader |
| src/apm_cli/deps/registry/client.py | New HTTP client for registry API (+ publish) |
| src/apm_cli/deps/registry/auth.py | New auth resolution + URL-prefix matching for lockfile replay |
| src/apm_cli/deps/registry/init.py | New registry package exports |
| src/apm_cli/deps/lockfile.py | Add registry fields; opportunistic lockfile v2 bumping; registry-aware to_dependency_ref |
| src/apm_cli/deps/installed_package.py | Carry registry resolution metadata to lockfile |
| src/apm_cli/deps/git_reference_resolver.py | Add semver-range selection over remote tags |
| src/apm_cli/core/experimental.py | Register marketplace_authoring + registries experimental flags |
| src/apm_cli/config.py | Add config helpers for registry url/token in config.json |
| src/apm_cli/commands/publish.py | New apm publish command to publish to registry |
| src/apm_cli/commands/config.py | Add registry.<name>.{url,token} config keys (gated) |
| src/apm_cli/cli.py | Register publish CLI command |
| docs/src/content/docs/reference/policy-schema.md | Document registry_source policy block |
| docs/src/content/docs/reference/manifest-schema.md | Document registries: and registry dependency forms |
| docs/src/content/docs/reference/lockfile-spec.md | Document lockfile v2 + registry fields + bumping rules |
| docs/src/content/docs/reference/experimental.md | List new experimental flags |
| docs/src/content/docs/reference/cli/install.md | Document registry install workflow |
| docs/src/content/docs/reference/cli/config.md | Document registry.* config keys + precedence |
| docs/src/content/docs/guides/registries.md | New registries guide |
| docs/src/content/docs/guides/private-registries.md | New private registries guide |
| docs/src/content/docs/getting-started/authentication.md | Add registry token docs |
| docs/src/content/docs/enterprise/security.md | Update security model for registries feature |
| docs/src/content/docs/enterprise/registry-proxy.md | Clarify proxy vs dedicated registries |
| docs/src/content/docs/consumer/manage-dependencies.md | Document registry dependency shapes (experimental) |
| docs/astro.config.mjs | Add sidebar entry for private registries |
| CHANGELOG.md | Add unreleased entry for registries feature |
Comments suppressed due to low confidence (3)
src/apm_cli/models/apm_package.py:1
_route_unscoped_to_default_registryroutes any shorthand with a#refto the default registry without validating that the selector is a semver value/range. That conflicts with the new semver-gate behavior asserted intests/unit/test_apm_yml_registries.py(e.g.,#main/SHA must be rejected when default routing applies). Fix by validatingdep.referencewith the registry semver gate prior to routing, and raise aValueErrorwith the existing remediation guidance (use explicit- git:object form) when the ref is not a valid semver selector.
src/apm_cli/models/dependency/reference.py:1_parse_registry_object_entryclaimsversionis 'strict semver, parse-time enforced' (docstring + module tests), but the implementation only checks non-empty string and does not validate semver/range. This will allow invalid refs (e.g. branch names) intosource='registry'deps and defer failure unpredictably. Fix by validatingversionusing the existing semver gate (apm_cli.deps.registry.semver.is_semver_range) and raising a clearValueErrorwhen it fails.
src/apm_cli/models/apm_package.py:1- The env-var hint does not sanitize
.in registry names (only-), but the actual token resolution logic maps both-and.to_(see_sanitized()indeps/registry/auth.py). This can mislead users for registries likecorp.main. Fix by applying the same normalization in this error message (or centralizing the formatting in a shared helper) so the suggested env var name matches runtime behavior.
98de01c to
b5656bd
Compare
|
Regarding the review "Comments suppressed due to low confidence (3)":
|
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 4 | 1 | Docs describe a sound registry resolver design with one semantic error (resolved_url as trust anchor) and a hash-mismatch retry that should be a hard fail. |
| CLI Logging Expert | 0 | 3 | 1 | Hash-mismatch severity and 401/403 remediation hint are underspecified; apm publish output contract has no example output or exit-code table. |
| DevX UX Expert | 0 | 3 | 2 | Solid feature gate + credential flow; three gaps: no cli/publish.md, silent git-dep rerouting hazard needs a callout, cli/experimental.md Available flags table missing registries. |
| Supply Chain Security Expert | 1 | 2 | 2 | Docs accurately scope the experimental feature; one critical behavior flaw (re-download on hash mismatch), two recommended gaps, two nits. |
| OSS Growth Hacker | 0 | 4 | 2 | Registries docs land the governance angle well but miss a hook sentence, a runnable quickstart, and a flywheel story for apm publish authors. |
| Auth Expert | 0 | 3 | 1 | Registry auth model is sound and well-separated; three documentation gaps could cause silent misconfigs or user confusion in production. |
| Doc Writer | 1 | 4 | 2 | guides/registries.md is absent from sidebar nav; significant content duplication between the two new guide pages; minor consistency issues. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 5 follow-ups
- [Doc Writer] (blocking-severity) Add
guides/registries.mdto sidebar nav indocs/astro.config.mjs-- Blocking defect in this PR: the page is unreachable via nav and must be fixed before merge. - [Supply Chain Security Expert] Clarify and document hash-mismatch behavior: hard abort vs re-download+warning -- If code already hard-fails, docs must be corrected here. If code re-downloads, open a code bug and add a security callout. The ambiguity must be resolved.
- [DevX UX Expert] Create
docs/reference/cli/publish.mdand addregistriestoexperimental.mdAvailable flags table -- apm publish is only documented inline; no reference page exists. experimental.md table is stale post-merge. Both are high-discoverability gaps. - [Auth Expert] Document name-sanitization collision (corp-main vs corp_main -> CORP_MAIN) and silent anonymous fallback risk -- Together these create a scenario where a mis-spelled env var silently degrades to anonymous access in CI with no warning.
- [OSS Growth Hacker] Add a runnable end-to-end quickstart to
guides/registries.md(60-second path toapm installfrom a private registry) -- Highest conversion lever for the enterprise persona; named integrations with no working example read as aspirational.
Architecture
classDiagram
direction TB
class RegistryConfig {
<<ValueObject>>
+name str
+url str
+is_default bool
}
class RegistryMap {
<<Strategy>>
+merge(policy, project, workspace, user) RegistryMap
+resolve_default() RegistryConfig
+lookup(name) RegistryConfig
}
class TokenResolver {
<<Strategy>>
+resolve(registry_name) AuthContext
}
class AuthContext {
<<ValueObject>>
+bearer str
+basic_user str
+basic_pass str
}
class RegistryResolver {
<<Strategy>>
+route(dep, registry_map) ResolvedDep
+resolve_version(dep, registry) str
+download_and_verify(dep) bytes
}
class PolicyEnforcer {
<<ChainOfResponsibility>>
+require list
+allow_non_registry bool
+check(dep, registry_map) None
}
class LockfileEntry {
<<ValueObject>>
+source str
+version str
+resolved_url str
+resolved_hash str
}
class Lockfile {
<<Aggregate>>
+lockfile_version str
+dependencies list
+bump_version_if_needed() None
}
class ExperimentalFlag {
<<Guard>>
+registries bool
}
class RegistryResolver:::touched
class RegistryMap:::touched
class TokenResolver:::touched
class PolicyEnforcer:::touched
class LockfileEntry:::touched
class Lockfile:::touched
class ExperimentalFlag:::touched
classDef touched fill:#fff3b0,stroke:#d47600
RegistryMap *-- RegistryConfig : contains
TokenResolver ..> AuthContext : returns
RegistryResolver ..> RegistryMap : lookups
RegistryResolver ..> TokenResolver : delegates auth
RegistryResolver ..> LockfileEntry : produces
PolicyEnforcer ..> RegistryMap : validates against
Lockfile *-- LockfileEntry : contains
ExperimentalFlag ..> RegistryResolver : gates
ExperimentalFlag ..> RegistryMap : gates
flowchart TD
A([apm install]) --> B[Read apm-policy.yml]
B --> C[Read project apm.yml]
C --> D[Read workspace apm.yml]
D --> E[Read config.json]
E --> F[Merge RegistryMap\nURL precedence: policy > project > workspace > config]
F --> G{Experimental flag\nregistries enabled?}
G -- no --> H[Git resolver only\nlockfile v1]
G -- yes --> I[PolicyEnforcer\nfail-closed if required registry missing]
I -- fail --> Z1([Exit 1: missing registry URL])
I -- pass --> J[For each DependencyEntry]
J --> K{Entry form?}
K -- git: object --> L[GitResolver: resolve commit SHA]
K -- path: object --> M[Local filesystem]
K -- shorthand with ref or id: object --> N{Default registry configured?}
N -- no --> Z2([Error: no default registry])
N -- yes --> O[RegistryResolver.route]
O --> P[TokenResolver: env var > config.json > anonymous]
P --> Q[GET /v1/packages/.../versions]
Q --> R[Select highest matching version]
R --> S[GET resolved_url: download archive]
S --> T{SHA-256 verify}
T -- mismatch --> V([Fail or re-download+warn\nsee follow-up 2])
T -- match --> W[Extract to apm_modules/]
W --> U[Write lockfile entry\nsource:registry, resolved_hash:sha256:...\nlockfile_version:2]
L --> X[Write lockfile entry\nresolved_commit:SHA40, lockfile_version:1]
U --> Y{allow_non_registry: false?}
X --> Y
Y -- blocked --> Z3([Exit 1: non-registry dep blocked])
Y -- pass --> AA([apm install complete])
Recommendation
Fix the sidebar nav entry (blocking doc-writer finding) before merge -- it is a two-line change in astro.config.mjs and there is no reason to ship an unreachable page. Separately, get a code-team confirmation on hash-mismatch behavior: if the code hard-fails today, update the docs in this PR; if it re-downloads+warns, add a security callout noting this is a known limitation and open a code-side issue. Once those two items are resolved, this PR is ready to ship with the remaining recommended follow-ups tracked as issues. The feature positioning is strong, the experimental gate is correctly scoped, and the governance story is exactly what the enterprise persona needs.
Full per-persona findings
Python Architect
- [recommended] resolved_url is documented as the trust anchor but it is not -- resolved_hash is at
docs/src/content/docs/guides/registries.md
guides/registries.md says "resolved_url is the trust anchor for re-installs". A trust anchor is the root of verification. resolved_url is only the re-fetch address; resolved_hash (SHA-256) is the actual integrity pin. If the registry URL migrates, the lockfile resolved_url breaks reproducibility even though the hash would still validate a correctly-fetched archive.
Suggested: "resolved_hash is the integrity anchor; resolved_url is the preferred re-fetch address. If the URL is unreachable, APM falls back to resolving via registry name + id + version." - [recommended] Hash mismatch triggers re-download + warning instead of a hard fail at
docs/src/content/docs/enterprise/security.md
Silently retrying a hash-mismatched download from the same server is unsafe: if the mismatch is caused by tampering at the registry, the retry fetches from the same tampered source. - [recommended] SHA-256 algorithm agility gap has no documented upgrade path at
docs/src/content/docs/guides/private-registries.md
The resolved_hash uses "sha256:..." prefix encoding which implies algorithm agility in the wire format, but docs state no upgrade path exists. The encoding capacity is already there.
Suggested: "The sha256: prefix in resolved_hash is algorithm-tagged by design; future releases may support sha384: or sha512: via a registry.(name).hash_algorithm config key." - [recommended] Transitive policy enforcement lacks a documented mechanism for registry-sourced transitive deps at
docs/src/content/docs/guides/registries.md
Both guides state policy checks apply transitively, but for registry-sourced deps, transitive deps are opaque unless the Registry HTTP API exposes a dependency manifest. Docs do not state whether APM recursively resolves them. - [nit] URL precedence table omits workspace ~/.apm/apm.yml from token chain at
docs/src/content/docs/guides/private-registries.md
CLI Logging Expert
- [recommended] Hash mismatch is documented as "a warning" but escalation path on repeated failure is not described at
docs/src/content/docs/enterprise/security.md
Suggested: "If the re-downloaded archive also fails hash verification, APM exits 1 with an [x] error naming the package and the expected vs actual digest." - [recommended] 401/403 remediation hint is described but never shown at
docs/src/content/docs/guides/private-registries.md
Suggested: Add terminal block: "[x] Registry auth failed for corp-main (HTTP 401). Set APM_REGISTRY_TOKEN_CORP_MAIN or run: apm config set registry.corp-main.token (token)" - [recommended] apm publish has no example output, no exit-code table, and "--verbose shows detailed output" is too vague at
docs/src/content/docs/guides/registries.md
Suggested: Add Output subsection with terminal blocks for success, --dry-run, 409, and 401/403. Add exit-code table (0=success, 1=upload/auth error, 2=usage error). - [nit] 409 Conflict surfaces an HTTP status code to users rather than an actionable APM-style message at
docs/src/content/docs/guides/registries.md
Suggested: "[x] Version 1.0.0 of acme/my-pkg already exists in corp-main. Bump the version in apm.yml and re-run apm publish."
DevX UX Expert
- [recommended] apm publish has no cli/publish.md reference page at
docs/src/content/docs/reference/cli/
Every APM command has a dedicated cli/(command).md page. A user running "apm publish --help" expecting a reference page will find nothing.
Suggested: Add docs/src/content/docs/reference/cli/publish.md with synopsis, options table, exit codes, and CI example. - [recommended] Default registry silently reroutes ALL shorthand deps -- no migration callout for existing projects at
docs/src/content/docs/guides/private-registries.md
When a user adds "default: corp-main" to an existing apm.yml, existing git deps like "acme/git-server#main" are silently rerouted. This violates the mental model that deps do not silently change resolution strategy.
Suggested: Add a caution callout with a before/after migration snippet showing the explicit "- git:" object form. - [recommended] reference/cli/experimental.md Available flags table omits registries at
docs/src/content/docs/reference/cli/experimental.md
Suggested: Add "registries" to the Available flags table in cli/experimental.md. - [nit] Option A / Option B labels imply mutual exclusivity but both can be used together at
docs/src/content/docs/guides/private-registries.md - [nit] apm experimental list example output is stale after this PR at
docs/src/content/docs/reference/experimental.md
Supply Chain Security Expert
- [blocking] Hash mismatch triggers re-download+warning instead of hard abort at
docs/src/content/docs/enterprise/security.md
A SHA-256 mismatch means the bytes are NOT what the lockfile recorded. Re-downloading from the same resolved_url does not help if the registry was compromised. Documenting "re-download and warn" trains users to treat integrity failures as transient noise rather than security events.
Suggested: hash mismatch -> fatal error, print expected vs actual SHA-256, adviseapm lock --refreshonly after manual investigation. - [recommended] resolved_url as re-fetch anchor is a MITM/substitution vector not addressed in docs at
docs/src/content/docs/reference/lockfile-spec.md
Suggested: "resolved_url changes on lock refresh indicate a registry-side move; treat as a supply-chain event requiring human review before merging the updated lockfile." - [recommended] allow_non_registry: false gap for local path deps and VCS deps not documented
If path: deps silently pass the gate, an attacker who controls a developer machine can inject a path dep that bypasses registry provenance. Docs must explicitly state the behavior. - [nit] SHA-256 only with no upgrade path -- docs should note this with a tracking issue reference
- [nit] Token env var NAME collision risk (e.g. NAME=DEFAULT) not documented
OSS Growth Hacker
- [recommended] guides/registries.md needs a one-line hook in the opening paragraph at
docs/src/content/docs/guides/registries.md
Suggested: Lead with: "APM registries give your org controlled sources, locked versions, and byte-level SHA-256 verification for every AI package -- without changing how developers run apm install." - [recommended] No end-to-end runnable example: reader cannot reach a working apm install within 60 seconds at
docs/src/content/docs/guides/registries.md
Suggested: Add a "Try it in 5 minutes" section with configure -> auth -> install -> verify flow. - [recommended] apm publish author flywheel is underdeveloped at
docs/src/content/docs/guides/registries.md
Suggested: Add a closing "Share packages with your org" call-to-action with link to publish reference. - [recommended] Artifactory/JFrog named but no compatibility matrix or trust signal at
docs/src/content/docs/guides/private-registries.md
Suggested: Add a "Tested registries" table; mark untested ones as "community reports welcome." - [nit] Planned features list with no timeline signal reads as vaporware -- link to tracking issues
- [nit] "Do not represent as SLSA-compliant" could be reframed as a positive forward path at
docs/src/content/docs/guides/registries.md
Auth Expert
- [recommended] Basic auth has no config.json fallback but precedence table implies symmetry with Bearer at
docs/src/content/docs/guides/private-registries.md
A reader who switches to Basic auth after storing a Bearer token in config.json will get anonymous access with no error.
Suggested: "Basic auth credentials are env-var-only; there is no config.json equivalent. config.json supports Bearer tokens only via registry.(name).token." - [recommended] Name-sanitization collision (corp-main and corp_main both resolve to CORP_MAIN) is undocumented at
docs/src/content/docs/getting-started/authentication.md
Suggested: "Registry names that differ only in hyphens vs underscores map to the same env var. Use distinct names to avoid credential sharing." - [recommended] Silent anonymous fallback can mask a mis-spelled env var name in CI at
docs/src/content/docs/guides/private-registries.md
Suggested: Add troubleshooting note: "If install succeeds but you expected authenticated access, verify the env var name matches the registry name derivation exactly." - [nit] APM_REGISTRY_PASS_{NAME} carries plaintext password; no CI masking guidance at
docs/src/content/docs/guides/private-registries.md
Doc Writer
- [blocking] guides/registries.md has no sidebar navigation entry -- the page is unreachable via the nav at
docs/astro.config.mjs
astro.config.mjs adds only "Private registries" to the Enterprise nav group. guides/registries.md is orphaned -- private-registries.md cross-links to it but users following the sidebar will never discover it.
Suggested: Add a "Registries" nav entry (slug: "guides/registries"). Consider whether both belong under Guides rather than Enterprise. - [recommended] Authentication, version selectors, and governance are duplicated verbatim between registries.md and private-registries.md at
docs/src/content/docs/guides/registries.md
Suggested: Make registries.md the single source of truth. Replace those sections in private-registries.md with a summary and cross-reference. - [recommended] Lockfile-spec "Working Draft" notice is inline prose, not a callout at
docs/src/content/docs/reference/lockfile-spec.md
Suggested: Replace with a Starlight :::note[Working Draft] callout scoped to registry fields only. - [recommended] Experimental caution title is inconsistent between the two new guide pages at
docs/src/content/docs/guides/private-registries.md
registries.md uses "[Experimental]"; private-registries.md uses "[Experimental -- dependency governance feature]". Standardize to "[Experimental]". - [recommended] "Private registries" placed in Enterprise nav group but page lives under guides/ -- mismatched placement at
docs/astro.config.mjs
Suggested: Move both registries nav entries to the same section (Guides or Enterprise). - [nit] guides/registries.md "See also" missing policy-schema link at
docs/src/content/docs/guides/registries.md - [nit] private-registries.md section 4.1 example has no disambiguation advice for git-routed deps at
docs/src/content/docs/guides/private-registries.md
Test Coverage Expert -- inactive
All 12 changed files are documentation files (CHANGELOG.md, docs/astro.config.mjs, docs/src/content/docs/); zero src//*.py files are touched.
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Note
🔒 Integrity filter blocked 2 items
The following items were blocked because they don't meet the GitHub integrity level.
- feat(registry): [FEATURE] Add package registry support for dependency… #1471
pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #1471
pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
To allow these resources, lower min-integrity in your GitHub frontmatter:
tools:
github:
min-integrity: approved # merged | approved | unapproved | noneGenerated by PR Review Panel for issue #1471 · ● 4.3M · ◷
99bd19b to
66a552d
Compare
|
I agree with @ghost - changes made:
|
0c53501 to
5a68337
Compare
|
@nadav-y thank you, let's fix ruff formatting and get this one in, great stuff! |
…fixes Add registry resolver, apm publish, apm outdated for registry deps, policy registry_source, and fail-closed install. Docs cover the HTTP API, private registries, review-panel follow-ups, and onboarding quickstart.
|
@danielmeppiel thanks 🙏 I've fixed ruff formatting and pushed. |
|
@nadav-y 2 failing integration tests in the merge queue: |
PR microsoft#1471 extended object-style dependency parsing to accept a 'registry' field alongside 'git' and 'path', and updated the ValueError message in DependencyReference.parse_from_dict to 'must have a git, path, or registry field'. Two pytest.raises regex assertions in test_runner_reference_phase3.py and test_runner_reference_resolution.py were left matching the old wording 'git or path field', causing the merge-queue Integration Tests Shard 3 job to fail with AssertionError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
|
@danielmeppiel |
…5 drift sweep) (#1511) * docs: backfill apm-usage and consolidate registry guides (v0.14->v0.15 drift sweep) Holistic docs-sync retrospective on the v0.14.0->v0.15.0 release window flagged 23 of 39 user-impact PRs as docs-debt: 7 Rule 4 violations (apm-usage/ skipped) plus 16 silent-drift PRs. This PR closes the highest-priority gaps (P0/P1 from the retrospective) in one sweep. Backfills (apm-usage/ training corpus): - dependencies.md: registry-sourced APM dep object form (#1471) - authentication.md: APM_REGISTRY_TOKEN_{NAME} precedence (#1471) - governance.md: registry_source + allow_non_registry policy (#1471) - package-authoring.md: apm publish workflow (#1471) and project-scope hook command path semantics (#1396) - commands.md: apm publish entry (#1471), apm config transport keys (#1308), apm compile live-reload + --clean --watch warning (#1403), Claude Code instruction dedup (#1146), MCP env-var placeholder resolution (#1277), AppLocker/WDAC staged-install diagnostic (#1390) Structural fix (per docs-impact-architect verdict): - Merge guides/private-registries.md INTO guides/registries.md with progressive disclosure (public -> private -> per-dep routing -> enterprise link). Adds Starlight redirect for the old slug, patches 5 cross-references across consumer/, reference/cli/. Editorial fixes (per editorial-owner sweep): - integrations/copilot-app.md (#1431): lead with user value before WS-IPC/SQLite mechanics; add 'restart the Copilot App once' troubleshooting hint - producer/compile.md: dedup the Claude Code instruction dedup explanation (was stated twice) - enterprise/security.md: reframe defensive memo voice ('do not call this X') to user voice ('here is what we provide / here is what we don't') Method: docs-sync skill end-to-end. 5-panelist fan-out plus CDO synthesis. Every CLI claim in the apm-usage adds was verified against the live 'apm <verb> --help' surface (S7 tool bridge). Out of scope (tracked as P1 follow-up): backfilling docs for the 16 silent-drift PRs grouped by subsystem (MCP, install, compile, auth). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: full-corpus regrounding audit (55 pages, 14 surgical fixes) Wave-batched grounding audit across 55 high-risk pages (CLI ref x27, schemas/specs x10, consumer ramp x12, onboarding x6). Each page's factual claims (flags, env vars, exit codes, schema fields, file paths, code links) was extracted and verified against current src/apm_cli/ and 'apm <verb> --help' output via S7 tool-bridge. Fixes applied (14 files): CLI reference: - pack.md: add --check-versions, --check-clean flags + exit codes 3, 4 - targets.md: expand copilot detection signals (5, not 1) - experimental.md: add copilot-app, marketplace-authoring, registries - install.md: dedup duplicate '## Exit codes' + '## Notes' sections Schemas / specs: - lockfile-spec.md: expand package_type enum to full 6-value list - manifest-schema.md: document plural 'targets:' alias (#1335) - environment-variables.md: add APM_BROAD_FETCH_DEPTH, APM_COPILOT_APP_DB - package-types.md: add 5th layout (hook_package, hooks/*.json only) Consumer ramp: - install-mcp-servers.md: fix stale code citation + 'Or' -> 'And' - private-and-org-packages.md: drop nonexistent BITBUCKET_APM_PAT Onboarding (6 broken navigation links, 4 files): - quickstart.mdx, getting-started/installation.md, getting-started/first-package.md, getting-started/migration.md: repoint self-loops and dead routes to actual page paths Process: dispatched as 6 parallel grounding-verifier agents (general- purpose) across disjoint page scopes; each agent had edit authority on its scope and applied surgical fixes inline. Reusable pattern via the docs-corpus-audit sibling skill design (PANEL + WAVE EXECUTION + S7 verifier fan-out, see files/docs-corpus-audit-design.md). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: wave 3 corpus audit + IA-reshuffle dead-link cleanup (53 pages) Second sweep of the regrounding audit. Covers the 57 pages deferred in wave 2: producer/ (15), enterprise/ (15), concepts/ (6), integrations/ (7), troubleshooting/ (7), contributing/ (3), reference tail (3), 404. Process: 6 parallel grounding-verifier agents on disjoint scopes; each agent extracts factual claims, S7-verifies against current source ('apm <verb> --help' + grep src/apm_cli/), and applies surgical edits inline. Same pattern as wave 2 (PANEL + WAVE EXECUTION + S7 verifier fan-out). Orchestrator post-pass swept three cross-corpus broken-link patterns the per-scope agents could not fix alone. High-signal factual fixes: enterprise/governance-guide.md: - --output-file -> --output (real flag is --output / -o) - 7+17 check count -> 8+17 (8 baseline checks, not 7) enterprise/apm-policy.md: - '16 of 22 checks' -> '17 of 25 checks' (phantom counts) - conflated --no-policy (install-only) with APM_POLICY_DISABLE (env) enterprise/apm-policy-getting-started.md: - dropped 'apm compile' from list of commands that run policy (compile enforces zero policy per governance-overview.md L57) enterprise/policy-reference.md: - compilation.target.allow: added copilot, gemini, vscode, windsurf, agent-skills (only 5 of 9 runtimes were listed) enterprise/registry-proxy.md: - 'apm marketplace add --branch main' -> '--ref main' (no --branch flag) enterprise/security-and-supply-chain.md: - 3 stale source line-number citations corrected producer/author-primitives/index.md: - legacy '.hook.md' extension -> '.json' (hook_integrator scans JSON) - removed nonexistent '.apm/commands/' subdirectory from layout example concepts/lifecycle.md: - 4 reference-page links all pointed at install/ (copy-paste) Cross-corpus IA-reshuffle dead-link cleanup (orchestrator pass): - introduction/* -> concepts/* (4 links across 2 files) - guides/ci-policy-setup/ -> enterprise/enforce-in-ci/ (8 links, 4 files) - guides/pack-distribute/ -> producer/pack-a-bundle/ (5 links, 4 files) - guides/dependencies/ -> consumer/manage-dependencies/ (1 link) - guides/agent-workflows/ -> contextual canonical (3 links, 3 files) - guides/install-and-use/mcp-servers/ -> consumer/install-mcp-servers/ (3) - guides/compilation/ -> producer/compile/ (1) - guides/prompts/ -> producer/author-primitives/prompts/ (2) - guides/drift-detection/ -> enterprise/drift-detection/ (1) enterprise/security.md side-fix: - 'apm unpack scheduled for removal in v0.14' -> drop version target (APM is 0.15.0 and unpack still ships marked DEPRECATED in --help). Upstream remediation (refresh deprecation timeline in source or remove the shim) tracked outside this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: close deferred items from corpus regrounding audit Closes the three items deferred from the v0.14->v0.15 docs-sync retrospective and the full-corpus regrounding waves (commits 4f00c2b, 242bb9e, b80da69): 1. apm unpack source-side deprecation timeline - src/apm_cli/commands/pack.py: 'will be removed in v0.14' -> 'will be removed in a future release'. Current version is 0.15.0; the v0.14 target had already passed. Docs were softened in wave 3; this mirrors the choice in source. - CHANGELOG.md: [Unreleased] Fixed entry. 2. Bucket-C silent-drift backfills (20 PRs, parallel triage) - 3 grounding-verifier subagents reviewed 20 of the 21 bucket-C PRs (#1477 excluded as test-flake fix, no doc surface). Verdicts: 17 ALREADY_COVERED or NO_DOC_SURFACE (verified honestly against wave 2-3 backfills, not manufactured), 3 BACKFILLED: - #1385 SSH dep user-from-URL: added supported-form row in docs/src/content/docs/consumer/manage-dependencies.md and bullet in apm-usage/dependencies.md. - #1434 Copilot App schema range [13,15] + warn-not-fail: rewrote the 'Schema compatibility' paragraph in docs/src/content/docs/integrations/copilot-app.md (was factually wrong, claimed [13,13] hard-fail). - #1440 Copilot file-based detection signals: added the four .github/{instructions,agents,prompts,hooks}/ directories to the canonical-signals list in troubleshooting/compile-zero-output-warning.md and to the apm-usage commands.md + package-authoring.md auto-detect rules. 3. docs-corpus-audit skill extracted - .apm/skills/docs-corpus-audit/SKILL.md: first-class skill module emitted from the genesis design artifact used to drive waves 2 and 3. Pattern: PANEL + WAVE EXECUTION + S7 verification. Wave-batched (scales as O(waves), not O(claims)), disjoint page ownership (no merge conflicts), orchestrator post-pass for cross-corpus drift patterns invisible to per-scope agents. - references/design-handoff.md: full design artifact preserved for future maintainers. - Sibling to docs-sync (per-PR), not a replacement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: fix dead links + address Copilot review findings Two classes of fix on PR #1511: 1. Deploy Docs CI -- starlight-links-validator failure (2 dead links) - getting-started/first-package.md:18 and quickstart.mdx:40 used absolute /apm/getting-started/installation/ paths introduced in wave 2 (242bb9e). Converted to relative paths matching the surrounding link convention. - Verified with local 'npm run build' under docs/: 'All internal links are valid.' 2. Copilot PR review -- 7 inline factual accuracy comments, all verified against source and addressed: - apm-usage/package-authoring.md: hook path rewrite is performed by 'apm install' (hook integrator pass), not 'apm compile'. - apm-usage/dependencies.md + docs/guides/registries.md: registry resolver requires semver per apm_cli/deps/registry/semver.py (is_semver_range gate). Removed examples implying opaque labels (#stable, #v2.0.0, 'latest') route through a registry; updated selector tables to flag non-semver refs as rejected for registry sources. - apm-usage/dependencies.md + docs/guides/registries.md: lockfile_version: '2' promotion triggers on registry deps OR git-source semver resolution fields (constraint / resolved_tag / resolved_at per lockfile.py:_needs_v2, issue #1488), not just registry deps. - apm-usage/authentication.md: 'token:' in apm-policy.yml is not parse-rejected, only surfaces as an 'Unknown top-level policy key' warning per policy/parser.py. Still discouraged (leaks to repo), but the rejection mechanism is different from apm.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill(docs-corpus-audit): refactor under genesis discipline + self-test Round-trip assessment found the original SKILL.md draft violated genesis SoC in 7 ways: 1. Invented inline 'grounding-verifier' persona instead of composing shared agent personas (python-architect for S7, doc-writer for edits). R3 EXTRACT in reverse. 2. Subagent prompt template inlined in SKILL body (~40 lines that belong in assets/). 3. IA-reshuffle grep patterns hard-coded in body as bash heredoc -- the patterns rot per release and belong in scripts/ with --help and a versioned update cadence. 4. PHANTOM DEPENDENCY on docs-sync's substrate (.apm/docs-index.yml, personas, panelist-return-schema, the apm-usage Rule-4 corpus) never declared via tool-call probes -- A9 SUPERVISED EXECUTION violation per genesis Step 7b. 5. Missing A8 ALIGNMENT LOOP: wave agents edited inline and nothing re-verified the edits grounded. 6. DISPATCH COLLISION risk vs docs-sync: identical 'drift between docs and code' triggers; dispatcher LLM could misroute. 7. BUNDLE LEAKAGE: references/design-handoff.md was session-history (maintainer-scope), not runtime-loaded. Per genesis 3.5 it must NOT ship with the user-facing bundle. Refactor: - SKILL.md (218 lines, well under 500-line cap): adds explicit Sibling Contract table with docs-sync; declares roster as composition of existing personas via relative links; PROBE / RISK-TRIAGE / WAVE / POST-PASS / ALIGNMENT-LOOP / COMMIT / PR phases; sharpened trigger description naming whole-corpus scope. - assets/subagent-prompt-template.md: extracted the per-scope prompt that composes python-architect + doc-writer. - assets/panelist-return-schema.json: explicit JSON schema for agent returns; orchestrator validates and rejects malformed. - scripts/scan-cross-corpus-drift.sh: deterministic cross-corpus drift sweep with 4 pattern groups (ia-links, stale-deprecation, absolute-base, ascii-leak). Non-interactive, --help-documented, stdout/stderr split per genesis script conventions. - evals/{trigger,content}-evals.json + README.md: ship gate exercising 10+10 trigger queries (docs-sync boundary is the load-bearing distinction) and 3 seeded-drift scenarios with control baselines. - Deleted references/design-handoff.md (bundle leak; design artifact stays in session state only). Self-test (proves the refactor works end-to-end): - Ran scan-cross-corpus-drift.sh against the live corpus; it immediately surfaced two genuine misses that wave 3 missed: - src/apm_cli/commands/pack.py:606: click help= string still said 'removed in v0.14' (the logger.warning at line 633 was fixed last commit; this is a sibling string the wave 3 agent didn't see because each agent only owned ~9 pages). - docs/src/content/docs/reference/cli/unpack.md:9: caution banner still said 'scheduled for removal in v0.14'. - Both softened to 'in a future release' (consistent with the rest of the wave 3 choice). - Lint clean; docs build clean ('All internal links are valid'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill(docs-grounding-verifier): claim-level grounding harness + 7 drift fixes New sibling skill to docs-corpus-audit. Genesis-designed PIPELINE-of-PANELS (RAGAS-faithfulness adapted from RAG to docs/code): - Stage 1: per-page LLM claim extraction - Stage 2: deterministic grep-based evidence retrieval (S7, no LLM) - Stage 3: adversarial LLM grounding judge (A7, 4-verdict calibrated) Empirical proof bundle (.apm/skills/docs-grounding-verifier/evals/runs/proof/): - 5 high-stakes pages -> 75 atomic claims extracted - Tally: 63 GROUNDED / 6 PARTIAL / 4 CONTRADICTED / 2 UNSUPPORTED (84%) - Trigger eval: 20/20 dispatch classification correct (precision=1.0, recall=1.0, specificity=1.0, pass_gate=true) High-confidence drift fixes applied: - apm-policy.md: MCP transport defaults (was 'block sse/streamable-http by default' -> actually allow=None means all permitted; sample policy now correctly framed as restriction example) - apm-policy.md: inheritance levels (was '5 levels including team policy' -> canonical chain is 3 semantic levels; 5 is MAX_CHAIN_DEPTH for intermediate extends: jumps) - Plus 5 editorial fixes from prior pass (examples, registries x2, security, copilot-app) Lower-confidence findings (judge retrieval gaps, vague reasoning) left for follow-up rather than risk introducing new drift via speculative edits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… resolution
Implements the full package registry feature — dedicated resolver, publish command, lockfile replay, and policy wiring — as described in the official feature request "[FEATURE] Add package registry support for dependency resolution".
feat(deps): dedicated registry resolver alongside the Git resolver
fix(registry): require published_at in VersionEntry to match spec
feat(publish): add apm publish command PUT /v1/packages/{owner}/{repo}/versions/{version} for producer push to the registry.
feat(registry): lockfile replay (npm install model) Honor apm.lock.yaml during apm install for registry-sourced deps. Bypass /versions, fetch from locked resolved_url, verify against locked resolved_hash. apm update bypasses replay.
feat(policy): plumb manifest registries: into registry-source check run_dependency_policy_checks now receives apm_package.registries from all four call sites (policy_gate, policy_target_check, run_policy_checks, run_policy_preflight). Closes the wiring gap where the fail-closed branch fired for every policy.require name regardless of apm.yml configuration. Adds transitive-enforcement tests and an integration tripwire at the policy_gate call site.
feat(registry): require registry+path in object form; defer @Shorthand to v2 Object form fields are now required. The acme/foo@corp-main#1.0.0 shorthand is deferred — @ collides with npm/cargo/pip version syntax, with SSH git@host, and with marketplace plugin shorthand.
Description
Brief description of changes and motivation.
Fixes # (issue)
Type of change
Testing