Skip to content

fix(policy): unify --policy / --policy-source forms across CLI help and docs#1000

Merged
danielmeppiel merged 2 commits intomicrosoft:mainfrom
edenfunf:fix/998-policy-source-help-ssot
Apr 28, 2026
Merged

fix(policy): unify --policy / --policy-source forms across CLI help and docs#1000
danielmeppiel merged 2 commits intomicrosoft:mainfrom
edenfunf:fix/998-policy-source-help-ssot

Conversation

@edenfunf
Copy link
Copy Markdown
Contributor

TL;DR

The "policy source" format accepted by apm audit --policy and apm policy status --policy-source is documented inconsistently across four locations — two Click help texts and two docs sections, with the parser supporting owner/repo shorthand that is missing from three of the four. This PR establishes a single Python constant as the source of truth, threads it through both Click decorators, aligns the docs, and adds six lockstep regression tests so future drift fails CI rather than going unnoticed for months.

Note

Closes #998 and #994. #994 reported a broken cross-reference (Same shape as 'apm install --policy' — a flag that does not exist); investigation revealed #998, the underlying drift. Both are fixed in one PR because they touch the same line of code; splitting would force the same line to be rewritten twice with no benefit.

Problem (WHY)

Why these matter: the per-flag accepted-forms list is a contract the CLI advertises and the parser enforces; today they describe the same contract differently in four places, which violates the discoverability guarantee the docs imply.

Approach (WHAT)

# Change
1 Define POLICY_SOURCE_FORMS_HELP in a new lightweight module (apm_cli/policy/_help_text.py) co-located with the parser.
2 Both apm audit --policy and apm policy status --policy-source Click decorators import and embed the constant via f-string.
3 The two cli-commands.md bullets are rewritten to the same canonical wording — this also subsumes #994, since the broken cross-reference is rewritten away rather than redirected to another command.
4 The parser docstring in discovery.py is corrected to list all five resolution branches (it was missing owner/repo) and points readers to the constant as the user-facing rendering.
5 Six lockstep tests (tests/unit/policy/test_help_consistency.py) pin the constant, both Click helps, both docs bullets, and the absence of apm install --policy references docs-tree-wide.

Implementation (HOW)

  • src/apm_cli/policy/_help_text.py (new) — holds POLICY_SOURCE_FORMS_HELP. Tiny, zero imports, lives next to the parser so a parser-behavior change forces a help-text update in the same diff.
  • src/apm_cli/commands/audit.py — imports the constant, uses f-string interpolation in the --policy Click help so the command-specific suffix (Used with --ci ... [experimental]) is preserved.
  • src/apm_cli/commands/policy.py — imports the constant, replaces the previous (already-most-accurate) help string with f"Override discovery. {POLICY_SOURCE_FORMS_HELP}".
  • src/apm_cli/policy/discovery.py — expands the discover_policy() docstring from 4 cases to 5 (the previous text omitted owner/repo) and cross-references the constant.
  • docs/src/content/docs/reference/cli-commands.md — the two affected bullets (lines 435 and 523) are rewritten to canonical wording mirroring the constant, using markdown backticks for inline forms.
  • tests/unit/policy/test_help_consistency.py (new) — six lockstep tests; mutation-tested to confirm each catches a distinct drift mode.

Trade-offs

  • Did not document the 3-segment host/owner/repo form. The parser supports it via _fetch_from_repo but no user-facing surface advertises it today. Promoting an internal capability to public API is a product decision, not a docs fix. Conservative call: keep the user-facing list at four forms; readers needing explicit-host can use the https:// URL form.
  • Did not change enterprise/policy-reference.md. That page already lists 4 forms correctly (different wording, same information). Pursuing byte-identical SSOT across audience-distinct docs would expand scope and trade reader-friendly prose for build-system parity.
  • Did not refactor discover_policy(). Behavior is unchanged; only the docstring was corrected and a cross-reference to the constant added. The 507 unit tests in tests/unit/policy/ pass unchanged.

Benefits

  1. Drift becomes a CI failure, not a months-later bug report. Mutation testing during review confirmed each lockstep test catches a distinct drift mode (constant edit, decorator bypass, docs deletion, broken cross-reference reintroduction).
  2. Discoverability of owner/repo shorthand. --policy and --policy-source users can now find this form from any single doc page or --help invocation.
  3. [cli-consistency] CLI Consistency Report — 2026-04-27 #994 closed without an intermediate state. The offending line is touched once and lands at the canonical wording — no "fixed cross-ref but still incomplete" middle stage.
  4. One-line maintenance. Future help-text changes for these flags require updating one Python constant, not four hand-maintained surfaces.

Validation

apm audit --help (relevant fragment):

  --policy TEXT                   Policy source. Accepts: 'org' (auto-discover
                                  from your project's git remote),
                                  'owner/repo' (defaults to github.com), an
                                  https:// URL, or a local file path. Used
                                  with --ci for policy checks. [experimental]

apm policy status --help (relevant fragment):

  --policy-source TEXT       Override discovery. Accepts: 'org' (auto-discover
                             from your project's git remote), 'owner/repo'
                             (defaults to github.com), an https:// URL, or a
                             local file path.

Lockstep regression tests:

$ pytest tests/unit/policy/test_help_consistency.py -v
test_canonical_constant_lists_all_supported_forms              PASSED
test_audit_policy_help_uses_canonical_constant                 PASSED
test_policy_status_help_uses_canonical_constant                PASSED
test_docs_audit_policy_bullet_lists_all_forms                  PASSED
test_docs_policy_status_bullet_lists_all_forms                 PASSED
test_no_broken_install_policy_cross_reference_anywhere_in_docs PASSED
============================== 6 passed in 0.45s ==============================
Aggregate regression on all touched code (563 tests)
$ pytest tests/unit/policy tests/unit/test_audit_policy_command.py \
    tests/unit/test_audit_ci_auto_discovery.py \
    tests/unit/commands/test_policy_status.py
============================= 563 passed in 2.33s =============================

The 26 failures observed in tests/unit/ outside this set are pre-existing on main (marketplace / install / cowork / config — none touched by this PR), confirmed by stashing the changes and rerunning the same selection on the parent commit.

Mutation testing performed during review

Each mutation was applied locally, the relevant test was run, then the mutation was reverted.

Mutation Tests caught it Result
Delete 'owner/repo' from POLICY_SOURCE_FORMS_HELP 3 (constant + audit help + policy status help) Caught
Replace audit.py help with hand-written string (bypass constant) 1 (audit help test) Caught
Reintroduce apm install --policy reference in governance-guide.md (a different docs page) 1 (docs-wide test, with file path in error) Caught
Delete owner/repo from cli-commands.md:523 bullet 1 (policy status doc bullet test, with bullet text in error) Caught

How to test

  • pytest tests/unit/policy/test_help_consistency.py -v — expect 6 passes.
  • apm audit --help — expect the --policy line to mention owner/repo, https://, and local file path.
  • apm policy status --help — expect the --policy-source line to mention all four canonical forms.
  • grep -r "apm install --policy" docs/ — expect no matches (closes [cli-consistency] CLI Consistency Report — 2026-04-27 #994).
  • Try apm policy status --policy-source contoso/.github — expect the parser to attempt fetch (not a parse-level rejection), confirming the newly-documented owner/repo form actually works.

…nd docs

Establish a single Python constant (POLICY_SOURCE_FORMS_HELP) as the
source of truth for the user-facing forms accepted by --policy /
--policy-source. Both Click decorators and both cli-commands.md bullets
align to it; six lockstep tests pin the relationship so future drift
fails CI. Also corrects the discover_policy() docstring, which had
been omitting the owner/repo case.

Closes microsoft#998 and microsoft#994. microsoft#994 (broken cross-reference to a non-existent
'apm install --policy' flag) is subsumed because the offending line is
rewritten to the canonical wording rather than redirected.
@danielmeppiel danielmeppiel enabled auto-merge April 28, 2026 05:27
@danielmeppiel danielmeppiel added this pull request to the merge queue Apr 28, 2026
Merged via the queue into microsoft:main with commit e232e98 Apr 28, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Policy source format documented inconsistently across CLI help and docs [cli-consistency] CLI Consistency Report — 2026-04-27

2 participants