PDX-508: enforce mustContainArgument best-practice rules in the local validator#207
Conversation
RCA: 21 required-argument rules (control-flow conditions, assertion comparison/expected values, BDD descriptions, SOQL/SQL/DB/REST connection+result args) shipped in provar_best_practices_rules.json with check.type "mustContainArgument", but the local engine never registered that check type — every one hit the silent-pass fallback and never fired, so external MCP users got none of these load/exec-blocking checks offline. Fix: implement validateMustContainArgument in bestPracticesEngine.ts and register it; it flags steps whose apiId matches check.apiId when the required check.argument is absent or an empty self-closing <argument/>, reusing the existing argumentHasValue/getCallArguments helpers for parity with the NitroX required-arg validators. Exact apiId match (plus :variant suffix), disabled steps skipped, nested control-flow steps checked, aggregated one-violation-per-rule with count. Adds 7 reproduce/cleared unit tests and documents the newly-enforced rule class in docs/mcp.md.
Quality Orchestrator🟢 LOW · 🧪 Tests to Run · Running 1 of 54 tests
▶ Run commandnpx vitest run \
unit/mcp/bestPracticesEngine.test.ts⚡ quality-orchestrator · |
There was a problem hiding this comment.
Pull request overview
Adds local/offline enforcement for mustContainArgument best-practice checks so that required-argument rules in provar_best_practices_rules.json no longer silently pass when Quality Hub is unavailable.
Changes:
- Implemented
validateMustContainArgumentand registered it inVALIDATOR_REGISTRYsomustContainArgumentrules are executed locally. - Added unit tests covering missing/empty required arguments, aggregation/counting, disabled-step skipping, apiId non-matches, and nested-step recursion.
- Documented the newly enforced
mustContainArgumentrule class indocs/mcp.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/mcp/tools/bestPracticesEngine.ts |
Adds and registers the mustContainArgument validator for local best-practice evaluation. |
test/unit/mcp/bestPracticesEngine.test.ts |
Adds targeted tests to ensure mustContainArgument behavior matches expectations and handles nesting/aggregation. |
docs/mcp.md |
Documents the offline/local enforcement of required-argument (mustContainArgument) rules and lists the currently enforced rule IDs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const requiredContainer = verdict.includes('UiWithRow') | ||
| ? 'UiWithRow' | ||
| : verdict.includes('UiWithScreen') | ||
| ? 'UiWithScreen' | ||
| : 'UiWithScreen'; | ||
| ? 'UiWithScreen' | ||
| : 'UiWithScreen'; |
RCA: Validated the local validator against quality-hub-agents MustContainArgumentValidator (best_practices_engine.py) and found four divergences: (1) empty <value/> or bare <value class="variable"/> was accepted but the backend treats present-but-empty as missing; (2) disabled steps were skipped but the backend checks them (a missing required arg is load-blocking regardless); (3) the legacy If/DoWhile condition-in-title format was not honored, risking false positives; (4) aggregating with count=N diverged the weighted-deduction score from the Lambda, which returns one violation per rule. Fix: add argumentHasMeaningfulValue mirroring the backend content checks (variable needs path/text, funcCall and gt/lt/eq/ne/ge/le/and/or/not operator classes count, compound needs parts, simple needs text); stop skipping disabled steps; honor the condition-in-title legacy exception for If/DoWhile; emit a single violation per rule with no count so the score stays in parity (message still names every offender); use exact apiId match. Extract callSatisfiesRequiredArg/stepLabel helpers to keep complexity under 20. Updates tests (12 cases) and docs/mcp.md to the backend-faithful semantics.
RCA: Copilot review flagged a nested ternary in validateUiActionNestingStructure where both non-UiWithRow branches return 'UiWithScreen', so the inner verdict.includes('UiWithScreen') test is dead and the expression is harder to read. Prettier reformatting in this branch surfaced the pre-existing line in the diff.
Fix: collapse to verdict.includes('UiWithRow') ? 'UiWithRow' : 'UiWithScreen' — behavior-preserving (the eliminated branch returned the same value) and clearer. No test impact; the nesting message tests still pass.
Backend parity validated against
|
Summary
mustContainArgumentcheck type in the local best-practices engine, unlocking 21 required-argument rules that shipped inprovar_best_practices_rules.jsonbut were silently skipped (no registered validator → silent-pass fallback). External/offline MCP users now get these load/exec-blocking checks without the Quality Hub back-end.provar_test_step_schema.json, the Validation Rule Registry doc, and new MCP resources are follow-up slices.What the validator does
apiCallwhoseapiIdmatchescheck.apiIdwhen the requiredcheck.argumentis absent or present only as an empty self-closing<argument/>with no<value>.getCallArguments/argumentHasValuehelpers so semantics match the NitroX required-argument validators exactly (no drift).:variantsuffix (NitroX convention); disabled steps skipped; steps nested inside control-flow containers are checked (recursive). One aggregated violation per rule withcount= offending step count (scored via the existing log2 formula).Rules now enforced: control-flow (
CONTROL-IF/WHILE/WAITFOR/FOREACH/SWITCH/SLEEP-*), assertions (ASSERT-COMPARISON-001,ASSERT-EXPECTED-001), BDD (BDD-GIVEN/WHEN/THEN-001), Apex/DB (SOQL-QUERY-001,DB-CONNECT-001/002,SQL-QUERY-001/002), web service (REST-CONN-001/002,REST-REQUEST-001).Jira
https://provartesting.atlassian.net/browse/PDX-508
Test plan
yarn compilepassesyarn test:onlypasses (1356 tests; 7 new reproduce/cleared tests formustContainArgument)node scripts/mcp-smoke.cjspasses (60/60)yarn lintpassesChanges
src/mcp/tools/bestPracticesEngine.ts: addvalidateMustContainArgument+ register it inVALIDATOR_REGISTRY.test/unit/mcp/bestPracticesEngine.test.ts: 7 tests — missing arg fires, populated clears, empty self-closing fires, count aggregation, disabled-step skip, non-matching apiId, nested-step recursion.docs/mcp.md: document the newly-enforcedmustContainArgumentrule class under Warning rules.Notes for reviewers / customer docs
docs/provar-mcp-public-docs.md/ the University course separately — this changes enforced offline behaviour and may warrant a mirror update.<argument/>(present but no<value>) as a violation, matching the sibling NitroX required-arg logic. If the QH back-end uses presence-only semantics formustContainArgument, flag it and I'll align.TOTAL_EXPECTEDor README changes were needed.