Skip to content

[BUG] apm audit --ci misses local-path sub-package MCP source drift (exits 0); direct MCP drift exits 1 #2127

Description

@danielmeppiel

Summary

apm audit --ci config-consistency does not detect when a locally-installed
local-path sub-package changes its MCP server config after install. If a
local-path sub-package's apm.yml MCP args (or command/env/transport) drift
away from the mcp_configs baseline recorded in apm.lock.yaml, audit exits
0 and reports "MCP configs match lockfile baseline". The equivalent direct
(root-declared) MCP config drift is caught and exits 1. The audit's
drift-detection guarantee has a false-negative blind spot for transitive
local-path MCP sources.

Baseline and environments

  • Baseline SHA: 4257698916736679478af4a9c37dc055b22b0d7a (origin/main); HEAD
    retained the behavior.
  • APM: 0.24.1 (uv run apm)
  • Platform / arch: macOS / arm64
  • Harness: MCP audit fixture target copilot (config comparison is
    target-independent)

Native contract

Exact reproduction

apm install --force --trust-transitive-mcp --target copilot --no-policy
apm audit --ci --no-policy --no-fail-fast -f json     # clean: exit 0

# mutate a local-path sub-package MCP config WITHOUT reinstalling
# edit packages/agent-config/apm.yml: MCP args ready -> changed
apm audit --ci --no-policy --no-fail-fast -f json     # observed: exit 0 (BUG)

# reconcile
apm install ... ; apm audit ...                        # exit 0 (lock updated)
# edit again: args changed -> changed-again
apm audit ...                                          # observed: exit 0 (BUG)

Expected and actual behavior

  • Expected: audit --ci fails (exit 1) and names the config difference when
    the current local sub-package MCP config differs from mcp_configs in
    apm.lock.yaml; reinstall reconciles and restores a clean audit.
  • Actual: on two distinct pre-reinstall source changes the lockfile args
    stayed old but audit exited 0 with "MCP configs match lockfile baseline".
    Reinstall updated the lock each time and the subsequent audit passed --
    confirming the drift was real and only the detection was missing.

Finder and independent confirmation

  • Finder: ogp-w4-macos (operations / governance / portability domain worker).
  • Confirmer: conf-ogp-r1-04 (distinct identity). Independently reproduced the
    local-path transitive drift exit 0 and the direct-MCP-drift exit 1 control
    on baseline and HEAD; confidence high.

Falsification controls

  • Clean control: unchanged local package, lock, and deployed config passed
    (exit 0).
  • Reproduction 1: local package MCP args ready -> changed while lock stayed
    ready -> audit exit 0 (miss).
  • Reconciliation control: reinstall updated lock to changed, audit passed.
  • Reproduction 2: args changed -> changed-again while lock stayed changed
    -> audit exit 0 (miss).
  • Inverse control: an equivalent direct MCP change exits 1 with "config
    differs from lockfile baseline" -> isolates the miss to the local-path
    sub-package source-derivation path, not audit's exit mapping.

Normalized fingerprint

CONFIG-CONSISTENCY:local-path-transitive-mcp-source-change-ignored:exit-0

Historical siblings

This directly matches the "fix-here, miss-there" hypothesis: the #2081 fix that
added provenance to suppress false positives is exactly the region that now
lacks post-install source re-derivation, producing a false negative.

Architecture diagnosis

Invariant (thematic): equivalent operations must derive their terminal signal
from a canonical result representing the full requested operation.

Canonical owner (this finding): src/apm_cli/policy/ci_checks.py:229-283
(_check_config_consistency). It derives current_configs only from
manifest.get_all_mcp_dependencies() (ci_checks.py:237-239), which
APMPackage implements as root production + root dev MCP entries only
(src/apm_cli/models/apm_package.py:525-563). The install path re-parses
installed sub-package apm.yml files, assigns dep.resolved_by, and stores the
merged config + provenance (src/apm_cli/integration/mcp_integrator.py:210-275,
src/apm_cli/install/mcp/integration.py:85-95,150-172). Audit never invokes
that source derivation, so a changed local-path sub-package is represented only
by the stale lock baseline. mcp_config_provenance is used at
ci_checks.py:256-265 solely to exempt lock-only transitive servers from
orphan reporting -- it is trusted as current instead of re-derived.

This is upstream verdict-input incompleteness, not a terminal-conversion
defect: src/apm_cli/policy/models.py:44-66 and
src/apm_cli/commands/audit.py:698 correctly aggregate CheckResult.passed
into CIAuditResult.passed and map to exit 0/1.

Confidence: confirmed for this single owner. Note (escalate): this finding and a
separate install-termination finding (#2126) rhyme under one "terminal-signal
parity" invariant but have no shared canonical owner (established by the
red-team architecture diagnosis, topology escalate). Fix each at its own
owner; do not build a cross-command consistency framework.

Blast matrix

  • command: audit --ci (direct vs local-path sub-package MCP parity;
    remote-sub-package regression). Siblings: update (rewrites mcp_configs +
    provenance), uninstall (orphan verdict), compile (source-vs-lock).
  • scope: project observed; global/user regression-relevant for MCP persistence
  • primitive: MCP observed (direct + local-path sub-package drift);
    hooks/instructions transitive source-vs-lock regression-relevant
  • target: copilot observed; claude/cursor/codex/windsurf/gemini/opencode
    regression-relevant (comparison must stay vendor-neutral, must not mutate
    native runtime files)
  • host: reachable local package observed; remote sub-package source
    regression-relevant
  • platform: macOS/arm64 observed; linux + windows regression-required

Fix constraints

Minimal remediation: define a side-effect-free, lock-bounded current-MCP-config
resolver that re-parses local-path sub-package manifests (reusing MCP dependency
parsing + root-first deduplication from MCPIntegrator without invoking install
I/O), and feed that complete current view into _check_config_consistency;
keep mcp_config_provenance for ownership/orphan classification only. No
harness translation layer. Required regression + mutation coverage: mutate
sub-package apm.yml command/args/env/transport without reinstall -> exit 1
naming the diff; unchanged transitive passes; direct drift still fails; #2081
clean-install non-orphan still protected; genuine unprovenanced orphan still
fails; duplicate server names keep root-first dedup; removing the transitive
source re-derivation (or substituting stored lock configs as current) must make
the drift regression fail. Fix/TDD/CI/mergeability ownership belongs to
batch-bug-shepherd.

Principles

Evidence

Finder, confirmer (two pre-reinstall drift reproductions + reconciliation and
direct-drift inverse controls), and the architecture diagnosis (cluster
arch-cluster-a-terminal-signal-parity, topology escalate) are archived by
the red-team run. Full sanitized artifacts available on request.


Filed by the APM red-team reliability sweep (Round 1). Confirmed by an
independent second reproduction. Handed to batch-bug-shepherd for triage, TDD,
review, CI, and mergeability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/audit-policyapm-policy.yml schema, custom_checks, install-time enforcement.area/lockfileLockfile schema, per-file provenance, integrity hashes, drift detection.status/needs-triageNew, awaiting maintainer review.type/bugSomething does not work as documented.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions