Repository Quality: Analyzer Catalog Hygiene Improvements

[Evangelink]

🎯 Repository Quality Improvement Report — Analyzer Catalog Hygiene

Analysis Date: 2026-05-11
Focus Area: Analyzer Catalog Hygiene (Custom)
Strategy Type: Custom — repository-specific focus on the MSTest Roslyn analyzer ecosystem

Executive Summary

This report covers the hygiene of the MSTest analyzer catalog — the set of files that define, document, and track the 63 diagnostic rule IDs (MSTEST0001–MSTEST0063). Three specific areas were found requiring attention:

1. A removed rule's constant was not commented out: MSTEST0039 (UseNewerAssertThrowsRuleId) is still declared as an active public const string in DiagnosticIds.cs, while the three other rules removed in Release 4.0.0 (MSTEST0006, MSTEST0034, MSTEST0053) are properly commented out with a "don't reuse this ID" note. This inconsistency leaves a risk of unintentional ID reuse.

2. Three typos in AnalyzerReleases.Shipped.md: Lines 19–21 record the category as "USage" instead of "Usage" for MSTEST0023, MSTEST0037, and MSTEST0045. While not functional, these represent inaccurate release documentation.

3. 71 TODO/FIXME comments in production source across 11 components, with the highest concentrations in Microsoft.Testing.Platform (18), MSTestAdapter.PlatformServices (18), and Microsoft.Testing.Extensions.TrxReport (14). Several of these are vague or even empty (// TODO: with no content), making them unactionable noise.

Full Analysis Report

Focus Area: Analyzer Catalog Hygiene

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total MSTEST diagnostic IDs	63 (MSTEST0001–MSTEST0063)	✅
Active diagnostic IDs	60	✅
Rules removed in 4.0.0	4 (MSTEST0006, MSTEST0034, MSTEST0039, MSTEST0053)	✅
Removed rules properly commented out in DiagnosticIds.cs	3 of 4	⚠️
Typos in AnalyzerReleases.Shipped.md	3 ("USage")	⚠️
Analyzer implementation files	54	✅
Suppressor implementation files	5	✅
TODO/FIXME comments in production src/	71	⚠️
Source files with TODO/FIXME	~55	⚠️
Empty/vague TODOs (e.g., // TODO:)	Several	❌

Findings

Strengths

• All active analyzer rules have implementations
• Help links are auto-generated consistently via DiagnosticDescriptorHelper pointing to learn.microsoft.com
• Removed rules are documented in AnalyzerReleases.Shipped.md under "Removed Rules" in Release 4.0.0
• The DiagnosticIds.cs commenting convention (with a "don't reuse" warning) is well-established for 3 of 4 removed rules

Areas for Improvement

MSTEST0039 constant not commented out (Medium severity)

In src/Analyzers/MSTest.Analyzers/Helpers/DiagnosticIds.cs:

// ✅ These are correctly commented out:
// public const string AvoidExpectedExceptionAttributeRuleId = "MSTEST0006"; - // Don't remove...
// public const string UseClassCleanupBehaviorEndOfClassRuleId = "MSTEST0034"; - // Don't remove...
// public const string AvoidAssertFormatParametersRuleId = "MSTEST0053"; - // Don't remove...

// ❌ This was also removed in 4.0.0 but is still active:
public const string UseNewerAssertThrowsRuleId = "MSTEST0039";
```

**Typos in release tracking document** (Low severity)

`src/Analyzers/MSTest.Analyzers/AnalyzerReleases.Shipped.md`, lines 19–21:
```
MSTEST0023 | USage | Info | Usage | Warning | ...   ← "USage" should be "Usage"
MSTEST0037 | USage | Info | Usage | Warning | ...   ← "USage" should be "Usage"
MSTEST0045 | USage | Info | Usage | Warning | ...   ← "USage" should be "Usage"

TODO/FIXME comments distribution in production source (Medium severity)

Component	TODO count
Microsoft.Testing.Platform	18
MSTestAdapter.PlatformServices	18
Microsoft.Testing.Extensions.TrxReport	14
MSTest.Analyzers	4
MSTest.SourceGeneration	3
MSTest.TestAdapter	3
Microsoft.Testing.Extensions.VSTestBridge	3
Other	8

Notable empty/vague TODOs:

• src/Platform/Microsoft.Testing.Platform/OutputDevice/SimplifiedConsoleOutputDeviceBase.cs: // TODO: (empty)
• src/Platform/Microsoft.Testing.Platform/OutputDevice/SimplifiedConsoleOutputDeviceBase.cs: // TODO (no colon, no content, two occurrences)

---

🤖 Suggested Improvement Tasks

Task 1: Comment out MSTEST0039 in DiagnosticIds.cs

Priority: Medium
Estimated Effort: Small

MSTEST0039 (UseNewerAssertThrowsRuleId) was removed in Release 4.0.0 but its constant in src/Analyzers/MSTest.Analyzers/Helpers/DiagnosticIds.cs is still active, unlike the three other rules removed in the same release. It should follow the same commenting convention:

// public const string UseNewerAssertThrowsRuleId = "MSTEST0039"; - // Don't remove this commented out line, and don't reuse this ID.

File: src/Analyzers/MSTest.Analyzers/Helpers/DiagnosticIds.cs, line 46

---

Task 2: Fix "USage" typos in AnalyzerReleases.Shipped.md

Priority: Low
Estimated Effort: Small

Three lines in the Release 4.1.0 "Changed Rules" section use USage instead of Usage for the New Category column. These should be corrected:

File: src/Analyzers/MSTest.Analyzers/AnalyzerReleases.Shipped.md, lines 19–21

-MSTEST0023 | USage | Info | Usage | Warning | DoNotNegateBooleanAssertionAnalyzer
+MSTEST0023 | Usage | Info | Usage | Warning | DoNotNegateBooleanAssertionAnalyzer
-MSTEST0037 | USage | Info | Usage | Warning | UseProperAssertMethodsAnalyzer
+MSTEST0037 | Usage | Info | Usage | Warning | UseProperAssertMethodsAnalyzer
-MSTEST0045 | USage | Info | Usage | Warning | UseCooperativeCancellationForTimeoutAnalyzer
+MSTEST0045 | Usage | Info | Usage | Warning | UseCooperativeCancellationForTimeoutAnalyzer

---

Task 3: Resolve or convert empty/vague TODO comments into tracked issues

Priority: Medium
Estimated Effort: Small

Several TODO comments have no description or are too vague to act on. Each should either be given a concrete description or replaced with a tracked GitHub issue reference. The following are the most egregious examples:

• src/Platform/Microsoft.Testing.Platform/OutputDevice/SimplifiedConsoleOutputDeviceBase.cs — multiple // TODO: and // TODO comments with no content
• Any // TODO: with no text after the colon

Search command: grep -rn "// TODO:\?\s*$" --include="*.cs" src/

For each empty TODO: either fill in a meaningful description, link to an existing issue, or remove it if no longer relevant.

---

Task 4: Audit and triage Microsoft.Testing.Platform and MSTestAdapter.PlatformServices TODO comments

Priority: Medium
Estimated Effort: Medium

The two components with the most TODO/FIXME comments (18 each) warrant a focused triage:

Microsoft.Testing.Platform TODOs include:

• CommandLineHandler.cs: "Replace Assembly with IAssembly" and "Replace RuntimeInformation with IRuntimeInformation" — these are architectural improvements that should be tracked
• CommonTestHost.cs: "Is this reachable? If so, are we missing await baseMessageBus.DisableAsync() here?" — potential bug
• ITestHostProcessInformation.cs: "As a follow-up, obsolete this property as error, and rename it..." — a deliberate planned deprecation that should be an issue
• TerminalOutputDevice.cs: "Replace this with proper CI detection that we already have in telemetry. Add CIConditionAttribute #5533 (comment)" — linked to an existing issue, could be removed once resolved

MSTestAdapter.PlatformServices TODOs include:

• TestDeploymentTests.cs: "This test has to have mocks" — test quality improvement
• TestMethodRunnerTests.cs: "Fix exception type"

Each TODO should be converted to a GitHub issue or removed if no longer relevant.

---

Task 5: Audit Microsoft.Testing.Extensions.TrxReport TODO comments

Priority: Low
Estimated Effort: Small–Medium

Microsoft.Testing.Extensions.TrxReport has 14 TODO/FIXME comments — the highest concentration in any single extension. This warrants a focused review to determine which represent actual planned work vs. outdated notes.

Run the following to inspect them all:

grep -rn "// TODO\|// FIXME" --include="*.cs" src/Platform/Microsoft.Testing.Extensions.TrxReport/

Each comment should be converted to a GitHub issue, documented with a decision not to fix, or removed.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type
2026-05-11	Analyzer Catalog Hygiene	Custom (first run)

---

🎯 Recommendations

Immediate Actions (This Week)

1. Comment out MSTEST0039 in DiagnosticIds.cs — Priority: Medium — 5-minute fix that prevents future ID reuse confusion
2. Fix "USage" typos in AnalyzerReleases.Shipped.md — Priority: Low — trivial correctness fix

Short-term Actions (This Month)

1. Remove or elaborate empty/vague TODO comments — Priority: Medium — improves codebase clarity
2. Triage MTP and PlatformServices TODOs — Priority: Medium — convert actionable items to issues
3. Audit TrxReport TODOs — Priority: Low — reduce noise in an extension with disproportionate TODO density

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-05-12 — Focus area will be selected based on diversity algorithm

Generated by Repository Quality Improver · ● 3.5M · ◷

• expires on May 13, 2026, 6:34 AM UTC