[tests] Improve NUnit runner reporting and dry-run auditing#11162
Open
simonrozsival wants to merge 3 commits intomainfrom
Open
[tests] Improve NUnit runner reporting and dry-run auditing#11162simonrozsival wants to merge 3 commits intomainfrom
simonrozsival wants to merge 3 commits intomainfrom
Conversation
Use NUnit's aggregate TestSuiteResult counts when reporting the final summary so skipped/ignored fixtures are included in the console output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes the Android NUnit harness’ final console summary counts so they match NUnit’s aggregate result tree (and therefore the XML output), addressing undercounting when entire fixtures/suites are ignored.
Changes:
- Update the runner to refresh
Passed/Failed/Skipped/InconclusivefromTestSuiteResultaggregate counts after adding each assembly’s results. - Compute
ExecutedTestsfrom the aggregate counters to keep the reported totals consistent.
simonrozsival
added
the
copilot
Add discovery-only dry runs, a noexclusions switch, explicit skip reasons, and ignored-test visibility for the Android NUnit harness. Also overwrite stale logcat captures so crash diagnosis reflects the current run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
simonrozsival
changed the title
simonrozsival
added a commit
that referenced
this pull request
Apr 20, 2026
Keep the trimmable typemap PR focused on the CoreCLRTrimmable lane and its targeted test-side plumbing. Move the generic NUnit runner and logcat auditing work to PR #11162, and drop the leftover Preserve-related noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
/review |
|
✅ Android PR Reviewer completed successfully! |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
🤖 Review Summary — ✅ LGTM (with minor suggestions)
What this PR does: Improves the NUnit test runner reporting by (1) fixing summary counts to use NUnit aggregate results, (2) adding dry-run discovery mode, (3) adding noexclusions=true for auditing, (4) surfacing excluded tests as Ignored with explicit reasons, and (5) overwriting logcat per run.
The overall approach is sound — moving from NUnit filter-based exclusions to RunState.Ignored with skip reasons is the right direction for auditing and observability.
Issues
| Severity | Count | Summary |
|---|---|---|
| 1 | TestNameMatches .Contains() checks are overly broad and could match unintended tests |
|
| 💡 Suggestion | 2 | Dead counter increments in TestFinished; repeated .Trim() calls |
Positive callouts
- Clean separation of concerns between
ApplyIgnoredExclusions,UpdateDiscoveredTestCounts, andApplyDryRunToMatchingTests - Good guard clauses in
AddExcludedCategory/AddExcludedTestName - The
noexclusions=truecorrectly bypasses only built-in exclusions while still honoring user-supplied extras — the right semantics for auditing - Logcat overwrite (
StdoutAppend = false) is a welcome improvement for debugging stale crashes
CI: Both public checks (license/cla, dotnet-android) are passing ✅
Generated by Android PR Reviewer for issue #11162 · ● 6.5M
- Fix TestNameMatches to use word-boundary-aware Contains checks (prefix with '.' or '+') instead of bare Contains that could match mid-word substrings - Remove dead counter increments in TestFinished since UpdateSummaryCounts overwrites them from the NUnit aggregate result - Extract static ParseBool helper to eliminate repeated Trim() calls in ProcessArguments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improve the Android NUnit harness so trimmable and test-lane investigations can audit discovery and exclusions reliably.
Changes
Why
These generic test-runner improvements were useful while working on #11091. I had problems identifying which tests were discovered, but did not run, as passed + skipped + failed < total.