Repository Quality: Collection Assertion API Completeness — Missing IsSubsetOf/IsNotSubsetOf and MSTEST0068 Coverage Gaps

[github-actions]

🎯 Repository Quality Improvement Report — Collection Assertion API Completeness

Analysis Date: 2026-07-22
Focus Area: collection-assertion-api-completeness
Strategy Type: Custom

Executive Summary

MSTest is in the middle of a large collection-assertion modernization wave: 136 new Assert.* Span/Memory overloads are staged in PublicAPI.Unshipped.txt, covering AreSequenceEqual, ContainsAll, AreAllDistinct, AreAllOfType, Contains, DoesNotContain, IsEmpty, IsNotEmpty, and more. The MSTEST0068 analyzer (CollectionAssertToAssert) was introduced to guide users from the legacy CollectionAssert class toward this modern API.

However, two CollectionAssert methods — IsSubsetOf and IsNotSubsetOf — have no counterpart in the modern Assert class, and the MSTEST0068 analyzer silently skips them (returns without a diagnostic). Users who call CollectionAssert.IsSubsetOf today receive no guidance, and are left relying on the legacy API indefinitely. Additionally, CollectionAssert.AreEquivalent and CollectionAssert.AreNotEquivalent overloads that accept an IComparer are also excluded from the analyzer migration path because the Assert.AreEquivalent/AreNotEquivalent methods use generic IEqualityComparer<T> rather than the non-generic IComparer.

Addressing these gaps will complete the CollectionAssert-to-Assert migration story and give users a fully modern, type-safe subset-testing API.

Full Analysis Report

Focus Area: Collection Assertion API Completeness

Current State Assessment

Metrics Collected:

Metric	Value	Status
New Span/Memory Assert overloads in Unshipped	136	✅ In progress
CollectionAssert methods handled by MSTEST0068	9 (AreEqual, AreNotEqual, AreEquivalent, AreNotEquivalent, AllItemsAreNotNull, AllItemsAreUnique, AllItemsAreInstancesOfType, Contains, DoesNotContain)	⚠️ Incomplete
CollectionAssert methods with NO Assert equivalent	2 (IsSubsetOf, IsNotSubsetOf)	❌ Gap
CollectionAssert overloads skipped due to comparer mismatch	4 (AreEquivalent + IComparer, AreNotEquivalent + IComparer, same generic variants)	⚠️ Gap
MSTEST0068 diagnostic on IsSubsetOf calls	0 (silently skipped)	❌ No guidance

Findings

Strengths

• The new Assert collection methods (AreSequenceEqual, ContainsAll, AreAllDistinct, etc.) have comprehensive IEnumerable(T) AND Span/Memory overloads, giving users full flexibility.
• Failure messages for the new assertions are detailed and structured (missing elements, null indices, mismatch positions surfaced).
• The MSTEST0068 analyzer covers 9 of the 11 distinct CollectionAssert method names.

Areas for Improvement

1. ⚠️ HIGH — Assert.IsSubsetOf and Assert.IsNotSubsetOf methods do not exist. CollectionAssert.IsSubsetOf verifies that every element of a subset collection appears in a superset. There is currently no Assert.IsSubsetOf(IEnumerable<T> subset, IEnumerable<T> superset) equivalent in the modern API. Users cannot migrate even if they want to.

2. ⚠️ HIGH — MSTEST0068 silently skips IsSubsetOf / IsNotSubsetOf. When the analyzer encounters CollectionAssert.IsSubsetOf(...) or CollectionAssert.IsNotSubsetOf(...), the switch expression returns null and the method returns immediately — no diagnostic, no migration hint. The user sees nothing.

3. ⚠️ MEDIUM — CollectionAssert.AreEquivalent(ICollection, ICollection, IComparer) has no Assert migration path. The condition !HasIComparerParameter(targetMethod) in CollectionAssertToAssertAnalyzer.cs:90 gates out the IComparer-bearing overloads of AreEquivalent/AreNotEquivalent. The modern Assert.AreEquivalent<T> uses IEqualityComparer<T>, not IComparer. Adding an IEqualityComparer-based migration note for these calls would help close the gap.

4. ⚠️ LOW — No SequenceOrder.InAnyOrder shorthand for AreSequenceEqual. The analyzer maps CollectionAssert.AreEquivalent (order-insensitive) to Assert.AreSequenceEqual(..., SequenceOrder.InAnyOrder). This is correct but verbose. A future Assert.AreEquivalentSequence or accepting SequenceOrder directly at the call level (already exists) is adequate — but documentation could clarify this path for migrators.

---

🤖 Suggested Improvement Tasks

The following actionable tasks address the findings above.

Task 1: Add Assert.IsSubsetOf and Assert.IsNotSubsetOf methods

Priority: High
Estimated Effort: Small

Add Assert.IsSubsetOf<T>(IEnumerable<T>? subset, IEnumerable<T>? superset, ...) and the corresponding IsNotSubsetOf overload to the Assert class in src/TestFramework/TestFramework/Assertions/, mirroring the IEnumerable<T> and non-generic IEnumerable patterns established by ContainsAll and AreAllDistinct. Include Span/Memory variants for consistency with the current wave of additions. Add entries to PublicAPI.Unshipped.txt (both net/ and base).

Relevant files:

• src/TestFramework/TestFramework/Assertions/ — add Assert.IsSubsetOf.cs
• src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt
• src/TestFramework/TestFramework/PublicAPI/net/PublicAPI.Unshipped.txt
• test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsSubsetOf.cs (new)

---

Task 2: Add IsSubsetOf / IsNotSubsetOf mappings to MSTEST0068 (CollectionAssertToAssert)

Priority: High
Estimated Effort: Small

Once Assert.IsSubsetOf exists (Task 1), extend the switch in CollectionAssertToAssertAnalyzer.cs to map:

• CollectionAssert.IsSubsetOf(subset, superset) → Assert.IsSubsetOf(subset, superset)
• CollectionAssert.IsNotSubsetOf(subset, superset) → Assert.IsNotSubsetOf(subset, superset)

The argument order matches (subset first, superset second), so a FixKindSwapTwoArgs variant or a new simple fix kind should work. Add a corresponding code fixer entry in CollectionAssertToAssertFixer.cs and unit tests in MSTest.Analyzers.UnitTests.

Relevant files:

• src/Analyzers/MSTest.Analyzers/CollectionAssertToAssertAnalyzer.cs
• src/Analyzers/MSTest.Analyzers.CodeFixes/CollectionAssertToAssertFixer.cs
• test/UnitTests/MSTest.Analyzers.UnitTests/CollectionAssertToAssertAnalyzerTests.cs

---

Task 3: Provide migration guidance for CollectionAssert.AreEquivalent(IComparer) overloads

Priority: Medium
Estimated Effort: Small

Currently, CollectionAssert.AreEquivalent(expected, actual, comparer) is silently skipped by MSTEST0068 because the IComparer non-generic parameter has no direct Assert.AreEquivalent<T>(IEqualityComparer<T>) equivalent. Two options:

• Option A (Recommended): Surface a diagnostic with an informational message saying the call cannot be auto-migrated because Assert.AreEquivalent<T> requires a generic IEqualityComparer<T> — guiding the user to refactor manually. This is better than silent skipping.
• Option B: If feasible, add CollectionAssert.AreEquivalent(ICollection, ICollection, IEqualityComparer) and wire up the migration automatically.

Relevant files:

• src/Analyzers/MSTest.Analyzers/CollectionAssertToAssertAnalyzer.cs (lines 90–91)

---

Task 4: Audit and document unmigrated CollectionAssert patterns in docs/migration guide

Priority: Medium
Estimated Effort: Small

Update the migration documentation (e.g., the CollectionAssertToAssert analyzer docs or the MSTest migration guide) to explicitly list which CollectionAssert methods:

• Have a one-to-one Assert equivalent and are auto-migrated by MSTEST0068
• Have a semantic equivalent but require manual argument or type changes
• Currently have no Assert equivalent (until Task 1 lands)

This prevents user confusion when MSTEST0068 fires on some calls but not others in the same test file.

Relevant files:

• docs/ or wiki (if docs live externally)
• src/Analyzers/MSTest.Analyzers/CollectionAssertToAssertAnalyzer.cs — XML doc for the diagnostic

---

Task 5: Add IEnumerable<T> overloads with IEqualityComparer<T> for ContainsSingle

Priority: Low
Estimated Effort: Small

The Assert.ContainsSingle method has IEnumerable<T> and predicate-based overloads but no IEqualityComparer<T> overload for the value-based form. All other comparison-based collection assertions (Contains, ContainsAll, DoesNotContain, AreAllDistinct) accept a comparer. Adding ContainsSingle<T>(T expected, IEnumerable<T> collection, IEqualityComparer<T> comparer, ...) would complete the pattern.

Relevant files:

• src/TestFramework/TestFramework/Assertions/Assert.ContainsSingle.cs
• src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type
2026-07-21	analyzer-code-fix-actionability-gap	Custom
2026-07-20	sourcegen-reflectionfree-coverage-gaps	Custom
2026-07-16	mstest-sdk-msbuild-property-reference-gap	Custom
2026-07-14	assertion-telemetry-coverage-gap	Custom
2026-07-10	cicd-workflow-hygiene	Standard
2026-07-09	extension-async-cancellation-completeness	Custom

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add Assert.IsSubsetOf / Assert.IsNotSubsetOf — Priority: High. These are the only two CollectionAssert methods that have no modern Assert equivalent, breaking the migration story. The implementation is straightforward given the existing ContainsAll pattern.

2. Wire MSTEST0068 for IsSubsetOf/IsNotSubsetOf — Priority: High. Once the Assert methods exist, the analyzer changes are a 2-line switch entry and a fixer registration.

Short-term Actions (This Month)

1. Address IComparer-bearing AreEquivalent overloads — Priority: Medium. At minimum, surface an informational diagnostic instead of silently returning, so users understand why that specific call isn't being flagged.
2. Add ContainsSingle with comparer — Priority: Low. Completes the comparer consistency pattern across all value-based collection assertions.

Next analysis: 2026-07-23 — Focus area selected based on diversity algorithm

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🤖 Automated content by GitHub Copilot. Generated by the Repository Quality Improver workflow. · 291.7 AIC · ⌖ 7.91 AIC · ⊞ 9.4K · [◷]( · ◷)

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repository-quality-improver.md@main

• expires on Jul 24, 2026, 10:47 PM UTC