feat: close audit-blindness gap for local .apm/ content via virtual self-entry + includes: manifest field

[danielmeppiel]

TL;DR

apm audit --ci is structurally blind to locally-deployed .apm/ content. This repo (microsoft/apm) ships 28 files into .github/, .claude/, and other workspace targets through implicit root-.apm/ integration -- and zero of them are covered by audit, content-integrity verification, or content scanning. The lockfile records per-file SHA-256 hashes that are never re-verified. Reported by an external CISO, surfaced by an external user noting the canonical repo's empty manifest, confirmed by panel review.

This issue closes the gap end-to-end in one PR, shipped in the next minor release (v0.10.0). No phasing, no deprecation curve, no breaking change.

Problem statement

APM has two parallel mechanisms for "local content" that produce identical artifacts (deployed files + per-file hashes) but land in two incompatible lockfile representations:

Aspect	Mechanism A (declared local-path deps)	Mechanism B (implicit root .apm/)
Trigger	apm.yml declares - ./packages/foo	<root>/.apm/ exists, no manifest entry
Lockfile home	lock.dependencies[key] -- full LockedDependency	lock.local_deployed_files (flat List[str]) + lock.local_deployed_file_hashes (flat Dict[str, str])
Audit visibility	Full -- every check iterates lock.dependencies	Zero -- no consumer reads local_deployed_files

Mechanism B was introduced in PRs #626, #644, #715 to remove stub-file friction for projects whose root .apm/ IS the package. The DX win is real and must be preserved. The audit blindness was an unintended side effect of the data-model split.

Verified blind spots (3 surfaces)

1. src/apm_cli/policy/ci_checks.py -- every check (lockfile-exists, deployed-files-present L106-137, no-orphans, ref-consistency, content-integrity L222-251, config-consistency) iterates only lock.dependencies.items(). Zero references to local_deployed_files.

2. src/apm_cli/policy/policy_checks.py:628-636 -- unmanaged-files check has identical blindness; would falsely flag mechanism B files as unmanaged if enabled.

3. src/apm_cli/security/file_scanner.py:62-66 -- scan_lockfile_packages has the same blindness. The content most consumed by AI agents (instructions, agents, skills) is the content never scanned.

"Security theater" in the lockfile

src/apm_cli/install/phases/post_deps_local.py:113-114 writes per-file SHA-256 hashes into lock.local_deployed_file_hashes on every install. No code path re-reads them to compare against live files. A developer can hand-edit any deployed instruction file post-install and apm audit --ci will not detect it. Hashes computed but never verified is bookkeeping presented as integrity verification.

Bundle leak risk

src/apm_cli/bundle/packer.py:122-126 (apm format) iterates lockfile.get_all_dependencies() with no is_dev filtering. src/apm_cli/install/exporters/plugin_exporter.py:471-476 (plugin format) DOES filter is_dev: true. Currently safe by accident because local_deployed_files lives on a separate field; once we synthesize a self-entry into lock.dependencies (the fix below), the apm format packer needs an explicit guard to avoid leaking root content into bundles.

Evidence in this repo

$ grep -c 'name:' apm.yml
1
$ python -c "import yaml; print(len(yaml.safe_load(open('apm.lock'))['local_deployed_files']))"
28
$ apm audit --ci
[+] lockfile-exists | No dependencies declared -- lockfile not required
[+] deployed-files-present | All deployed files present
[...all green, 28 files invisible...]

The CISO's complaint is correct. The external user's complaint is correct. We have an audit hole AND a credibility-gap manifest in our own canonical repo.

Why this matters (product reasoning)

APM's positioning is "the governance layer for AI agent configuration -- complete audit trail for everything AI agents see." The MANIFESTO calls out "Reliable over Magic" and "Transparent processes that teams can audit, understand, and trust."

A canonical repo that deploys 28 files with no manifest declaration and no audit coverage violates this on three counts:

1. The audit story is hollow. We tell enterprises "run apm audit --ci in CI to detect drift" while the data structure those checks iterate is empty for our most common deployment shape (single-package projects).
2. The manifest doesn't reflect intent. A CISO reading our apm.yml in a PR diff sees nothing. Then they look at .github/agents/ and see 28 files appearing from nowhere. Trust is broken.
3. No incumbent has a unified audit story for AI tooling. This is APM's moat. A false version of the claim is worse than no claim.

The fix below makes the claim true.

The fix (single PR, ships v0.10.0, not phased)

Engine: virtual self-entry at lockfile read boundary

LockFile.from_yaml() synthesizes a virtual LockedDependency from existing local_deployed_files data and injects it into self.dependencies. to_yaml() extracts it back out before serialization to keep the YAML format stable. Every existing audit check, content scanner, and packer filter gains coverage through existing lock.dependencies iteration.

# src/apm_cli/deps/lockfile.py -- inside from_yaml(), after existing loads:
_SELF_KEY = "."
if lock.local_deployed_files:
    lock.dependencies[_SELF_KEY] = LockedDependency(
        repo_url="<self>",
        source="local",
        local_path=".",
        is_dev=True,
        deployed_files=list(lock.local_deployed_files),
        deployed_file_hashes=dict(lock.local_deployed_file_hashes),
    )

# Inside to_yaml() -- extract before serialization:
_self_dep = self.dependencies.pop(_SELF_KEY, None)
data["dependencies"] = [dep.to_dict() for dep in self.get_all_dependencies()]
if _self_dep:
    self.dependencies[_SELF_KEY] = _self_dep  # restore in-memory
    data["local_deployed_files"] = sorted(_self_dep.deployed_files)
    if _self_dep.deployed_file_hashes:
        data["local_deployed_file_hashes"] = dict(sorted(_self_dep.deployed_file_hashes.items()))

Critical invariants on the synthesized entry:

• is_dev: true -- prevents plugin bundle leakage (plugin_exporter.py already filters this).
• source: "local" -- the apm-format packer needs a 2-line guard to skip source: "local" entries.
• local_path: "." -- canonical key; downstream code can disambiguate.

Manifest: includes: field

New top-level optional manifest field. Default is auto (preserves the implicit .apm/ discovery DX win from #626/#644/#715). Explicit path lists supported for organizations that need governance.

# Default behavior -- auto-discover root .apm/ content.
includes: auto

# OR explicit governance:
includes:
  - .apm/agents/code-reviewer.agent.md
  - .apm/skills/apm-review-panel/SKILL.md
  - .apm/instructions/python.instructions.md

auto is never deprecated. It is the right default for the 95% case. Enterprise strictness is layered on via policy (next item), not forced on every project.

Policy: requires_explicit_includes knob

apm-policy.yml schema gains a boolean knob:

# apm-policy.yml
require:
  explicit_includes: true   # mandates includes: <list>, rejects includes: auto

Organizations needing CISO-grade governance set this in their org policy chain. Everyone else keeps auto.

Audit: hash verification + content scanning + manifest consent

1. _check_content_integrity in ci_checks.py -- now actually re-reads files and compares against deployed_file_hashes. Currently the hashes exist but no check verifies them.
2. _check_no_orphans -- exclude the synthetic . key from manifest comparison: if dep_key == ".": continue.
3. _check_lockfile_exists -- treat non-empty local_deployed_files as has_deps=True so the aggregate runner doesn't short-circuit on local-only repos.
4. New check _check_includes_consent -- if includes: is missing from manifest AND local_deployed_files is non-empty, emit a [!] warning advising the user to declare includes: auto. This is the explicit-consent surface for enterprise governance without forcing migration.
5. scan_lockfile_packages in file_scanner.py -- gains coverage automatically through the synthesized entry; add test coverage to lock in the behavior.

Packer: source: "local" guard

src/apm_cli/bundle/packer.py (around L122-126) -- skip entries with source == "local" in the apm-format packer, mirroring the existing rejection of local-path manifest deps at L89-97. Plugin format already correct via is_dev filter; add regression test.

Canonical repo: update microsoft/apm apm.yml

name: apm
version: 0.10.0
description: APM (Agent Package Manager) -- ship and govern AI agent context
author: Microsoft
license: MIT

# Local .apm/ content is auto-discovered and integrated into workspace
# targets (.github/, .claude/, etc.) on `apm install`. Declare 'auto' to
# opt in to implicit discovery; list paths for explicit governance.
includes: auto

dependencies:
  apm: []
  mcp: []

scripts: {}

This closes the external-user complaint and serves as the canonical example.

Acceptance criteria

All in one PR. All in v0.10.0.

Code

• LockFile.from_yaml() synthesizes virtual self-entry when local_deployed_files is non-empty (is_dev: true, source: "local", local_path: ".", repo_url: "<self>").
• LockFile.to_yaml() extracts the self-entry before serialization; YAML format is byte-stable for existing lockfiles (round-trip test).
• _check_lockfile_exists treats non-empty local_deployed_files as has_deps.
• _check_no_orphans excludes the synthetic . key.
• _check_content_integrity re-reads files and verifies hashes against deployed_file_hashes (currently does not verify).
• New _check_includes_consent warning when includes: missing and local_deployed_files non-empty.
• scan_lockfile_packages covers the self-entry (verified by test, not just by inheritance).
• apm.yml schema supports includes: auto and includes: [<path>, ...] (explicit list).
• apm-policy.yml schema supports require.explicit_includes: bool; policy gate rejects includes: auto when set.
• bundle/packer.py apm-format packer skips source: "local" entries.
• microsoft/apm apm.yml updated with includes: auto (separate commit in same PR).

Tests

• Lockfile round-trip preserves YAML format for existing mechanism-B lockfiles.
• apm audit --ci on this repo (post-fix) surfaces all 28 locally-deployed files.
• Hash mismatch test: hand-edit a deployed file, audit detects drift.
• apm pack (apm format) does NOT include self-entry files in the bundle.
• apm pack --format plugin does NOT include self-entry files in the bundle.
• includes: auto parses; includes: [<paths>] parses; includes: missing emits consent warning.
• Policy require.explicit_includes: true blocks install when manifest has includes: auto or no includes:.
• Self-entry NOT exposed via manifest.get_apm_dependencies() (the synthesis is a lockfile concern, not a manifest concern).

Docs

• docs/src/content/docs/reference/lockfile-spec.md -- document self-entry convention (the . key, <self> repo_url, is_dev: true invariant).
• docs/src/content/docs/reference/manifest-spec.md (or equivalent) -- document includes: field with auto and explicit-list forms.
• docs/src/content/docs/governance-guide.md §14 Known gaps and limitations -- remove the "no drift detection on deployed files" entry; it's no longer true.
• Policy reference docs -- document require.explicit_includes.
• Skill resource files -- update packages/apm-guide/.apm/skills/apm-usage/governance.md and any others affected by the schema change.

CHANGELOG

### Added
- New `includes:` manifest field. Declares how local `.apm/` content is
  discovered and integrated. `auto` (default) preserves existing implicit
  discovery; explicit path lists enable enterprise governance. Policy gains
  `require.explicit_includes` to mandate explicit declarations org-wide.
- `apm audit --ci` now verifies SHA-256 hashes of locally-deployed files
  against the lockfile, detecting post-install drift.

### Fixed
- `apm audit --ci` now covers locally-deployed `.apm/` content (agents,
  skills, instructions, hooks, commands). Previously, files deployed from
  root `.apm/` were tracked in the lockfile but invisible to all audit
  checks and the content-integrity scanner. Lockfile schema is unchanged;
  coverage is gained through internal read-boundary normalization.
- `apm pack` (apm format) no longer risks including locally-deployed root
  `.apm/` content in published bundles. Plugin format was already safe via
  `is_dev` filtering; both formats now have explicit test coverage.

Migration line: None. auto is the default. Existing projects work without changes. Audit may now correctly flag drift that was previously invisible -- this is a fix, not a break.

Out of scope (intentionally)

• Deprecating auto. Never. It's the right default. Enterprise strictness layers via policy.
• New manifest surface beyond includes:. No selfDependencies:, no provides:, no primitives:. The architectural panel debated and rejected these.
• Touching _integrate_root_project() or the install pipeline phase ordering. The fix is purely at the lockfile read/write boundary plus three audit-check tweaks. Install behavior is byte-identical.
• Migrating mechanism A (declared local-path deps). Already audited; not affected.

Naming rationale

includes: was chosen after rejecting:

• primitives: -- internal jargon, no external developer knows the term.
• provides: -- implies publishing semantics; this is about local integration.
• agents: -- too narrow; field covers instructions, skills, hooks, commands, agents.
• bundles: -- collides with apm pack bundle terminology.
• local: -- vague.
• selfDependencies: -- conflates "I ship X" with "I depend on X."

includes: is familiar from CSS @include, C #include, cargo include = [...]. Reads naturally: "This project includes these local AI capabilities."

Decision provenance

Crisis panel review (2026-04-23) with three independent specialist assessments:

• Python Architect: diagnosed dual-track as accidental structural duplication; recommended virtual self-entry at lockfile read boundary (~35 lines, 2 files); rejected new manifest surface as DX regression.
• Supply-Chain Security Expert: confirmed "security theater" (hashes computed but never verified); demanded six-item GA bar; required is_dev: true on self-entry to prevent bundle leakage.
• DevX UX Expert: tested three personas (npm/pip user, CISO, plugin author); argued implicit detection fails the CISO sniff test; proposed new manifest field with phased deprecation.
• OSS Growth Hacker: side-channelled "not either/or" reframe -- engine + story are separable concerns.
• APM CEO: arbitrated -- UX wins the principle (manifest declaration matters), Architect wins the default (preserve .apm/ DX), Security wins the GA bar (hard blockers ratified). Single-release scope per maintainer direction.

File map for the implementer

File	Change
src/apm_cli/deps/lockfile.py	Self-entry synthesis in from_yaml(); extraction in to_yaml().
src/apm_cli/policy/ci_checks.py	Fix _check_lockfile_exists gating; exclude . from _check_no_orphans; add hash verification to _check_content_integrity; add _check_includes_consent.
src/apm_cli/manifest/schema.py (or equivalent)	Add includes: field (auto / list-of-paths).
src/apm_cli/policy/policy_schema.py (or equivalent)	Add require.explicit_includes: bool.
src/apm_cli/install/phases/policy_gate.py	Wire explicit_includes enforcement.
src/apm_cli/bundle/packer.py	Skip source: "local" entries in apm-format packer (~L122-126).
src/apm_cli/security/file_scanner.py	No code change required; coverage gained via synthesis. Add test.
apm.yml	Add includes: auto; bump version to 0.10.0.
docs/src/content/docs/reference/lockfile-spec.md	Document . self-entry convention.
docs/src/content/docs/reference/manifest-spec.md	Document includes: field.
docs/src/content/docs/governance-guide.md	Remove §14 drift-detection gap entry.
packages/apm-guide/.apm/skills/apm-usage/governance.md	Update for new schema.
CHANGELOG.md	Add entries above.

Risk assessment

• Lockfile YAML stability: Round-trip test required. Synthesis happens purely in-memory; YAML format unchanged.
• is_dev: true correctness: If wrong, plugin bundles leak root content. Test required.
• Packer regression: If source: "local" guard is missing, apm bundles leak root content. Test required.
• Backward compat: auto default means no existing project breaks. New audit findings are correct exposure of pre-existing drift, not a regression.

Definition of done

A reviewer can say: "This repo's apm audit --ci now detects drift on every file under .github/, .claude/, and any other workspace target. Hand-editing a deployed agent file is caught. The manifest declares includes: auto. The CISO can read apm.yml and trust it. Plugin authors keep zero-ceremony .apm/ magic. Enterprise orgs can require explicit declarations via policy. CHANGELOG, docs, and skill resources reflect the new reality. All in one PR. Shipped in v0.10.0."