perf(install): memoize discovery, drop per-file resolve, expand skip dirs (#1533)

[danielmeppiel]

TL;DR

Fixes #1533 — apm install on awd-cli and large monorepos was 30–40x slower than necessary because discover_primitives re-walked the project tree once per (integrator, target) pair, and find_primitive_files did Path.resolve() per file. Three independent root causes, all fixed.

Scenario	Before	After	Speedup
awd-cli T=1 integrate phase	3.7s	0.80s	4.6x
awd-cli T=7 multi-target install	19.7s	0.76s	26x
Kubernetes discover (cold)	205s	5.4s	38x
TypeScript discover (cold)	297s	7.0s	42x
Warm discovery (cache hit)	n/a	26us	22500x

Problem (WHY)

apm install on this repo with one integration target took ~5s wall (Python startup dominant) but with 7 targets blew up to ~20s. On Kubernetes/TypeScript the discover phase alone took 200–300s. The master formula was N_walks = 3 * N_targets * (N_packages + 1) — the integrate phase looped per (integrator, target), each iteration re-walking the entire tree. Within each walk, find_primitive_files did path.resolve().relative_to(base.resolve()).as_posix() PER FILE — two lstat-per-component chains per file across 80k+ files on Kubernetes.

Approach (WHAT)

Three independent fixes plus instrumentation and an opt-in harness.

1. Discovery memoization: module-level cache keyed on (realpath(base), sorted_exclude_tuple). Cleared at top of run_install_pipeline. Cuts 9 walks/install → 3 unique walks.
2. Hot-path stat reduction: pre-split glob patterns ONCE per call; string-slice rel_root from os.walk root + base prefix instead of Path.resolve() per file; defer Path() construction until after a pattern matches.
3. Skip-dir expansion: added vendor, third_party, Pods, bower_components, jspm_packages, .gradle, target, .next, .nuxt, .cache, .turbo to DEFAULT_SKIP_DIRS.

Implementation (HOW)

flowchart LR
  A[run_install_pipeline] -->|reset| B[perf_stats.reset]
  A -->|reset| C[clear_discovery_cache]
  A --> D[integrate phase loop]
  D --> E[discover_primitives base=root]
  E -->|cache miss| F[find_primitive_files]
  E -->|cache hit| G[return shared collection]
  F -->|record_walk| H[perf_stats]
  E -->|record_discovery| H
  A -->|verbose-only| I[perf_stats.render_summary]

Loading

New module src/apm_cli/utils/perf_stats.py — process-scoped counters; reset(), record_walk(), record_discovery(), render_summary(logger, project_root). O(1) per walk overhead (one perf_counter + list append). Failures in render are surfaced as [!] warnings, not silently swallowed.

Verbose perf output (--verbose):

[#] Perf: 9 walks, 72 file matches, 6222 files visited, 0.739s total walk time
[#] Perf:   .: 3 walk(s) (726ms, 6108 files visited, 72 matched)
[#] Perf:   apm_modules/_local/apm-review-panel: 3 walk(s) (4ms, 36 files visited, 0 matched)
[#] Perf:   apm_modules/_local/batch-bug-shepherd: 3 walk(s) (9ms, 78 files visited, 0 matched)
[#] Perf: discovery: 9 call(s) (3 unique base(s), 6 cache hit(s), 66%)

Paths are relativized to project_root. Uses the [#] (metrics) STATUS_SYMBOLS entry from utils/console.py.

Opt-in perf harness tests/perf/ — 4 scenarios (Kubernetes discover, TypeScript discover, awd-cli install, multi-target breadth T=7). Skipped unless PYTEST_PERF=1 is set; clones large external repos to /tmp/perf-atlas-clones/ once per session. Not run on CI.

Test isolation tests/conftest.py — new autouse fixture clears _DISCOVERY_CACHE and resets perf_stats around every test so process-scoped globals do not leak across tests that exercise discover_primitives directly.

Removed dead code: deleted duplicate _glob_match at discovery.py:396 (segment-aware version at line 543 was the actually-called definition; old fnmatch version was unreachable shadow).

Trade-offs

• Cache scope is process-global, not per-InstallContext. Today the CLI is one-shot (fresh process per invocation). Multi-pipeline-in-one-process is handled by the autouse fixture for tests. Migrating to context-scoped storage requires threading InstallContext into 20+ callers — separate PR if concurrent installs ever become real.
• Identity-shared PrimitiveCollection across cache hits. Verified no caller mutates the returned collection in the integrate path; python-architect signed off.
• DEFAULT_SKIP_DIRS may surprise users with primitives in target/ (e.g. deployment-target specs). Mitigation deferred; debug logging is in place when a dir is pruned.

Validation evidence

• Lint (CI mirror): ruff check + ruff format --check + pylint R0801 + lint-auth-signals.sh all GREEN.
• Targeted tests: tests/unit/primitives/ (148) + tests/benchmarks/test_scaling_guards.py (minus pre-existing flaky TestComputePackageHashScaling::test_scaling_ratio) — all pass.
• Full suite: 25,477 pass / 234 skipped. 20 failures are pre-existing test-order-sensitive environmental flakes (TempDir, MCP registry import, copilot-app e2e) that pass in isolation and are NOT introduced by this change. The pre-existing TestComputePackageHashScaling::test_scaling_ratio flake (also fails on stashed clean tree) is separately tracked.
• Live install measurements: verified on this worktree via PYTHONPATH=src python -m apm_cli.cli install --verbose — all numbers in the TL;DR table reproduced from this branch.

How to test

# 1) Run the full unit suite (CI does this).
pytest tests/

# 2) Run the opt-in perf scenarios locally (requires clone access).
PYTEST_PERF=1 pytest tests/perf/ -v

# 3) Verify verbose perf output on a real install.
PYTHONPATH=src python -m apm_cli.cli install --verbose 2>&1 | grep '\[#\] Perf:'

Review process

Spawned three reviewers (performance-expert, python-architect, cli-logging-expert) per the apm-review-panel pattern. All returned fold_then_ship. Folded findings:

• delete dead duplicate _glob_match (perf, blocking)
• add autouse conftest fixture for cache isolation (arch)
• relativize perf paths via project_root, replace <unknown> with ., add [#] status symbol, add cache hit-rate percent, surface render errors as [!] (logging)
• drop the redundant _is_readable precheck on the hot path (perf — kept the function for its other 3 callers)

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[danielmeppiel]

Shepherd status: ready for maintainer review

PR state: MERGEABLE (no conflicts), BLOCKED due to code-owner approval gate.

CI summary (commit bdcbb895)

All 13 standard checks green: Lint, Build & Test (Shard 1/2), Analyze (actions/python), CodeQL, Coverage Combine, Spec conformance, PR Binary Smoke, APM Self-Check, NOTICE Drift, gate (the only required check per main-protection ruleset), license/cla.

One red check, not caused by this PR

PR Review Panel / apm (default, default, 0, microsoft/apm#main, false) fails with:

The template is not valid. .github/workflows/pr-review-panel.lock.yml (Line: 1087, Col: 32): A sequence was not expected

The offending expression is ${{ secrets.* }} on main. Verified pre-existing: same workflow fails identically on danielmeppiel/turbo-waddle and camilo/fix-opencode-tools-conversion. This PR does not touch .github/workflows/.

Remaining gates (both human / maintainer)

1. CODEOWNERS approval (require_code_owner_review: true, require_last_push_approval: true).
2. Merge queue uses ALLGREEN grouping; the PR Review Panel infra fix on main is a prerequisite for queue entry.

Shepherd loop terminal. No further automation steps will move this PR.