[Lean Squad] feat(ci): Task 9 — FV docs validation workflow + updated REPORT.md by Evangelink · Pull Request #7936 · microsoft/testfx

Evangelink · 2026-04-29T16:27:00Z

Add .github/workflows/fv-docs-validation.yml: a CI workflow that validates
the structural integrity of formal-verification/ artifacts on every PR or
push that touches formal-verification/**.

Checks performed:

Required top-level FV docs present (RESEARCH.md, TARGETS.md,
CORRESPONDENCE.md, CRITIQUE.md, REPORT.md)
Lean project files present (lakefile.toml, lean-toolchain, README.md)
lean-toolchain is non-empty and well-formed
🔬 Lean Squad disclosure present in top-level docs
specs/ directory exists
Phase-2+ targets in TARGETS.md each have an informal spec in specs/
(keyword-based matching handles CamelCase name decomposition)

Also update formal-verification/REPORT.md: add run history for the
2026-04-27 runs, expand target table with new targets, and document the
Lean toolchain blocker.

�� Lean Squad — automated formal verification agent
Run: https://github.com/microsoft/testfx/actions/runs/24986088168

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Fixes #7861

Add .github/workflows/fv-docs-validation.yml: a CI workflow that validates the structural integrity of formal-verification/ artifacts on every PR or push that touches formal-verification/**. Checks performed: - Required top-level FV docs present (RESEARCH.md, TARGETS.md, CORRESPONDENCE.md, CRITIQUE.md, REPORT.md) - Lean project files present (lakefile.toml, lean-toolchain, README.md) - lean-toolchain is non-empty and well-formed - 🔬 Lean Squad disclosure present in top-level docs - specs/ directory exists - Phase-2+ targets in TARGETS.md each have an informal spec in specs/ (keyword-based matching handles CamelCase name decomposition) Also update formal-verification/REPORT.md: add run history for the 2026-04-27 runs, expand target table with new targets, and document the Lean toolchain blocker. �� Lean Squad — automated formal verification agent Run: https://github.com/microsoft/testfx/actions/runs/24986088168 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds CI validation for the formal-verification/ artifact structure and updates FV reporting to reflect recent runs and current blockers, helping keep Lean Squad formal-verification documentation consistent over time.

Changes:

Introduce a GitHub Actions workflow that validates required FV docs, Lean project scaffolding, and basic TARGETS/specs consistency on PRs/pushes affecting formal-verification/**.
Update formal-verification/REPORT.md with refreshed status/summary and additional run history entries.

Show a summary per file

File	Description
`formal-verification/REPORT.md`	Updates FV status/summary text and adds 2026-04-27 run history entries.
`.github/workflows/fv-docs-validation.yml`	New CI workflow to validate presence/shape of FV docs, Lean project files, and Phase-2+ informal spec alignment.

Copilot's findings

Files reviewed: 2/2 changed files
Comments generated: 3

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Copilot's findings

Files reviewed: 2/2 changed files
Comments generated: 3

Copilot · 2026-04-30T07:23:12Z

+**Phase**: Early — Research expanded to 7 targets; informal specs extracted for `ArgumentArity` (merged) and `CommandLineParser.TryUnescape` (PR open). CI automation maturing. Lean toolchain blocked in CI runner; Task 3+ deferred.

 ## Summary

-The Lean Squad has surveyed the `microsoft/testfx` codebase and identified **seven** high-quality FV targets. Two informal specifications have been extracted. The Lean toolchain cannot currently be installed in the runner environment due to network restrictions.
+The Lean Squad has surveyed the `microsoft/testfx` codebase and identified **seven** high-quality FV targets in the command-line infrastructure and filter logic of Microsoft.Testing.Platform. All targets are pure or near-pure functions with clear algebraic properties, making them suitable for Lean 4 formal verification.

 Key findings:
- **`ArgumentArity`**: informal spec extracted (14 properties, 4 groups). Constructor does not enforce `Min ≤ Max`; `CommandLineOption` enforces this on construction, acting as the real guard.
+- **`ArgumentArity`**: informal spec extracted (14 properties, 4 groups). Constructor does not enforce `Min ≤ Max`; `CommandLineOption` enforces this on construction, acting as the real guard. Correspondence documented.
 - **`CommandLineParser.TryUnescape`**: informal spec extracted (24 properties, 5 groups, 2 confirmed bugs for single-char quote inputs). PR open.
 - **New targets identified**: `ResponseFileHelper.SplitCommandLine` (pure tokeniser) and `TreeNodeFilter.MatchFilterPattern` (Boolean algebra — ideal for structural induction proofs of De Morgan, double negation, idempotence).


The status/summary bullets appear out of date with the current FV artifacts: TARGETS.md lists several Phase-2 targets beyond just ArgumentArity and TryUnescape (e.g., ParseOptionAndSeparators, SplitCommandLine, TreeNodeFilter.MatchFilterPattern), and formal-verification/specs/ already contains additional *_informal.md files. Please update this section to reflect the actual set of extracted informal specs (or adjust phases elsewhere so the report is consistent).

Copilot · 2026-04-30T07:23:13Z

+      - name: Check FV 🔬 disclosure in top-level docs
+        run: |
+          FAILED=0
+          for doc in RESEARCH.md TARGETS.md CORRESPONDENCE.md CRITIQUE.md REPORT.md; do
+            if ! grep -q "🔬" "formal-verification/${doc}"; then
+              echo "::error file=formal-verification/${doc}::FV document missing 🔬 Lean Squad disclosure: ${doc}"
+              FAILED=1
+            else
+              echo "✅  formal-verification/${doc} has 🔬 disclosure"
+            fi
+          done
+          if [ "$FAILED" -ne 0 ]; then
+            exit 1
+          fi


This step fails the workflow if any required doc is missing the 🔬 disclosure marker, but the PR description/issue states the disclosure check should be warning-only (non-blocking). Consider emitting ::warning (and not exiting nonzero) so the workflow behavior matches the documented intent.

Copilot · 2026-04-30T07:23:13Z

+          TOOLCHAIN="$(tr -d '\r\n' < formal-verification/lean/lean-toolchain)"
+          if [ -z "$TOOLCHAIN" ]; then
+            echo "::error file=formal-verification/lean/lean-toolchain::lean-toolchain file is empty"
+            exit 1
+          fi
+
+          if [[ "$TOOLCHAIN" =~ [[:space:]] ]]; then
+            echo "::error file=formal-verification/lean/lean-toolchain::lean-toolchain must not contain whitespace"
+            exit 1
+          fi


The lean-toolchain validation strips all newlines (tr -d '\r\n'), so a malformed multi-line lean-toolchain file would be concatenated and could still pass the whitespace/regex checks. If the goal is to enforce a single well-formed toolchain spec, check that the file contains exactly one non-empty line (and fail if there are extra lines or trailing whitespace) before validating the pattern.

Evangelink

Summary

Workflow: Expert Code Reviewer
Date: 2026-04-30
Repository: microsoft/testfx

Key Findings

The two changed files are a new GitHub Actions workflow (.github/workflows/fv-docs-validation.yml) and a documentation update (formal-verification/REPORT.md). No production C# code is changed.

Correctness issues in the CI workflow — all in the final Python step:

Missing spec check emits ::warning:: instead of ::error:: (Line 141) — This is already silently misfiring: TreeNodeFilter.MatchFilterPattern is a Phase-2 target in TARGETS.md but has no corresponding file in formal-verification/specs/. The workflow reports success. The purpose of this step — to gate PRs that forget to add informal specs — is completely unenforced. Change to ::error:: + sys.exit(1).
Spec glob uses *.md instead of *_informal.md (Line 123) — The prior step counts *_informal.md files, but the validation step considers all .md files in specs/. Any incidental Markdown (README, notes) would satisfy a match, hiding a genuinely missing spec.
Vacuous all() on empty keyword list (Line 132) — For any target whose name decomposes entirely into tokens shorter than 3 characters, all(kw in s for kw in []) returns True for every spec file, so the target appears validated regardless of actual coverage.

Positive Observations

The workflow correctly scopes permissions to contents: read.
The shell steps for checking file existence, lean-toolchain format, and 🔬 disclosure are solid and correctly use exit 1 to fail CI.
Hardcoded loop variables in shell steps eliminate any injection risk.

Recommendations

Fix the three Python-step correctness bugs before merging. The most critical is converting the missing-spec diagnostic from ::warning:: to a hard failure; the other two address false-positive scenarios.

Generated by Expert Code Reviewer

🧠 Reviewed by Expert Code Reviewer 🧠

Evangelink · 2026-04-30T07:26:52Z

+              parts = re.sub(r'([a-z])([A-Z])', r'\1_\2', name)
+              return [w.lower() for w in re.split(r'[._\s]+', parts) if len(w) >= 3]
+
+          all_specs = [os.path.basename(p) for p in glob.glob(os.path.join(specs_dir, "*.md"))]


[Correctness] glob.glob(... "*.md") collects all Markdown files in specs/, not just *_informal.md files. Any incidental .md file (e.g., a future README.md or NOTES.md placed in that directory) would be treated as a candidate spec, producing false-positive matches.

Impact: The step directly above counts only *_informal.md files; this step silently widens the net, creating an inconsistency that could hide a genuinely missing informal spec.

Suggestion: Narrow the glob to match the established naming convention:

all_specs = [os.path.basename(p) for p in glob.glob(os.path.join(specs_dir, "*_informal.md"))]

Fixed in 39ac0ad — changed glob.glob(... "*.md") to glob.glob(... "*_informal.md") so the validation is consistent with the count step and won't match incidental markdown files.

Evangelink · 2026-04-30T09:11:53Z

@copilot address review comments

- Narrow glob from *.md to *_informal.md to prevent false-positive matches - Add empty-keywords guard to skip targets whose name has only short tokens (avoids vacuous all() returning True on empty list) - Change ::warning:: to ::error:: + sys.exit(1) so missing specs block CI - Remove redundant keywords[0] guard (now handled by the empty-keywords check) Agent-Logs-Url: https://github.com/microsoft/testfx/sessions/950eb316-fac6-464f-8197-0054dfa30e41 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot · 2026-04-30T09:17:27Z

@copilot address review comments

All three correctness issues addressed in 39ac0ad:

*.md → *_informal.md in the spec glob
Empty-keywords guard added before all() match
Missing spec diagnostic upgraded from ::warning:: to ::error:: + sys.exit(1)

Copilot AI review requested due to automatic review settings April 29, 2026 16:27

Copilot started reviewing on behalf of Evangelink April 29, 2026 16:27 View session

Copilot AI reviewed Apr 29, 2026

View reviewed changes

Comment thread .github/workflows/fv-docs-validation.yml Outdated

Comment thread .github/workflows/fv-docs-validation.yml Outdated

Comment thread .github/workflows/fv-docs-validation.yml Outdated

Evangelink mentioned this pull request Apr 29, 2026

[Lean Squad] Formal Verification Status #7793

Open

Apply suggestions from code review

b9f0085

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 30, 2026 07:19

Copilot started reviewing on behalf of Evangelink April 30, 2026 07:20 View session

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Evangelink commented Apr 30, 2026

View reviewed changes

Copilot started work on behalf of Evangelink April 30, 2026 09:12 View session

Copilot finished work on behalf of Evangelink April 30, 2026 09:18

Evangelink merged commit 485bb51 into main Apr 30, 2026
2 checks passed

Evangelink deleted the lean-squad/task9-fv-docs-ci-2026-04-27-fcf8710c1c8b58c4 branch April 30, 2026 09:29

Evangelink mentioned this pull request May 4, 2026

[docs] Update glossary - weekly full scan #8015

Open

Conversation

Evangelink commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Evangelink left a comment

Choose a reason for hiding this comment

Summary

Key Findings

Positive Observations

Recommendations

Uh oh!

Uh oh!

Evangelink Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Evangelink commented Apr 30, 2026

Uh oh!

Copilot AI commented Apr 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Evangelink commented Apr 29, 2026 •

edited

Loading