Repository Quality: Diagnostic & Error Message Quality Improvements

[Evangelink]

🎯 Repository Quality Improvement Report — Diagnostic & Error Message Quality

Analysis Date: 2026-05-08
Focus Area: Diagnostic & Error Message Quality (Custom)
Strategy Type: Custom — Invented for this testing framework's unique needs

Executive Summary

As a testing framework, MSTest's most critical user-touchpoints are the messages shown when tests fail or when the framework itself encounters errors. A thorough analysis of error messages across the codebase reveals several actionable opportunities: CollectionAssert and StringAssert APIs lack the [CallerArgumentExpression] support that Assert already has, leaving test failure messages less informative than they could be. Resource files used for localization have very low translator comment coverage (under 10%), impairing translation quality. A handful of cryptic or empty error messages (e.g., "err", "Invalid runsettings") can stump developers when they hit rare code paths.

Fixing these issues would improve the developer experience for both MSTest users writing tests and developers extending the platform. The changes are largely mechanical and low-risk, but their impact on day-to-day test debugging and platform extensibility is significant.

Full Analysis Report

Focus Area: Diagnostic & Error Message Quality

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total source .cs files	2,787	✅
Total exception throws with inline string literals	200	⚠️
CollectionAssert methods with [CallerArgumentExpression]	0 / many	❌
StringAssert methods with [CallerArgumentExpression]	0 / many	❌
Translator comments in PlatformResources.resx	19 / 199 (9.5%)	❌
Translator comments in FrameworkMessages.resx	7 / 95 (7.4%)	❌
Translator comments in Resource.resx (adapter)	10 / 108 (9.3%)	❌
Translator comments in MSTest.Analyzers/Resources.resx	5 / 171 (2.9%)	❌
Cryptic/placeholder error messages identified	3+	❌
Typos in user-visible resource strings	1 confirmed	⚠️

Findings

Strengths

• Assert methods (AreEqual, IsNull, IsTrue, AreSame, etc.) have excellent [CallerArgumentExpression] coverage, producing rich failure messages showing the exact expressions evaluated.
• Assert.ThrowsException/Assert.Throws also supports [CallerArgumentExpression], so action expressions appear in failure messages.
• Assert.AreEqual for strings provides diff output (AreEqualStringDiffFailMsg) — very useful for debugging.
• The Assert.That(Expression<Func<bool>> condition) overload automatically extracts the predicate expression for readable failures.
• Most user-facing messages are centralized in .resx resource files, enabling localization.

Areas for Improvement

1. [High] CollectionAssert and StringAssert lack [CallerArgumentExpression] — These widely-used assertion classes produce failure messages that don't capture which variables were under test, unlike the modern Assert methods. Users see "Element not found in collection" without knowing which collection or element variable was involved.

2. [High] Near-zero translator comment coverage in resource files — With 9.5% or less coverage, translators lack the context needed to produce accurate localized messages. The .xlf files generated for localization inherit this gap.

3. [Medium] Cryptic or placeholder error messages:

   • src/Platform/Microsoft.Testing.Platform/ServerMode/JsonRpc/Json/JsonObjectSerializer.cs:14 — throws InvalidOperationException("err") — a placeholder message that was never replaced.
   • src/Adapter/MSTestAdapter.PlatformServices/RunConfigurationSettings.cs:155 — throws FormatException("Invalid runsettings") — no indication of what is invalid or where.
   • src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/DotnetMuxerLocator.cs:401 — throws InvalidOperationException("Invalid image") — no context about what image or what was invalid.

4. [Low] Typo in PlatformResources.resx — Line 127: "The same instance of 'CompositeExtensonFactory' is already registered" — missing 'i': should be "CompositeExtensionFactory".

---

🤖 Suggested Improvement Tasks

The following actionable tasks address the findings above.

Task 1: Add [CallerArgumentExpression] to CollectionAssert methods

Priority: High
Estimated Effort: Medium

CollectionAssert (in src/TestFramework/TestFramework/Assertions/) currently has no [CallerArgumentExpression] attributes on any of its public methods. The modern overloads of Assert.AreEqual, Assert.IsNull, etc. all use this attribute, so the gap here is noticeable.

For example, in CollectionAssert.Membership.cs, the Contains(ICollection? collection, object? element, string? message) method should gain a [CallerArgumentExpression(nameof(collection))] string collectionExpression = "" parameter and a [CallerArgumentExpression(nameof(element))] string elementExpression = "" parameter. The failure message should prepend the expression info, mirroring the pattern in Assert.IsNull.cs.

Files to update: CollectionAssert.Membership.cs, CollectionAssert.Equality.cs, CollectionAssert.Equivalence.cs, CollectionAssert.Subset.cs, CollectionAssert.Type.cs.

---

Task 2: Add [CallerArgumentExpression] to StringAssert methods

Priority: High
Estimated Effort: Small

StringAssert (in src/TestFramework/TestFramework/Assertions/StringAssert.cs) has no [CallerArgumentExpression] attributes. All methods take value and substring parameters but never surface the original expressions in failure messages.

Add [CallerArgumentExpression(nameof(value))] string valueExpression = "" and [CallerArgumentExpression(nameof(substring))] string substringExpression = "" parameters to all public Contains, StartsWith, EndsWith, Matches, and DoesNotMatch overloads. Prepend expression context to failure messages following the same pattern used in Assert.IsNull.cs (CallerArgumentExpressionSingleParameterMessage or a two-parameter variant).

---

Task 3: Replace cryptic error messages with descriptive ones

Priority: Medium
Estimated Effort: Small

Three locations have placeholder or near-meaningless error messages:

1. src/Platform/Microsoft.Testing.Platform/ServerMode/JsonRpc/Json/JsonObjectSerializer.cs:14

   // Current:
   ? throw new InvalidOperationException("err")
   // Suggested:
   ? throw new InvalidOperationException($"The '{nameof(Properties)}' delegate on '{GetType().Name}' has not been initialized. Ensure all serializer instances are fully configured before use.")

2. src/Adapter/MSTestAdapter.PlatformServices/RunConfigurationSettings.cs:155

   // Current:
   throw new FormatException("Invalid runsettings");
   // Suggested:
   throw new FormatException($"The runsettings XML document has an invalid format. Ensure the XML is well-formed and matches the expected runsettings schema.");

3. src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/DotnetMuxerLocator.cs:401

   // Current:
   return archType ?? throw new InvalidOperationException("Invalid image");
   // Suggested:
   return archType ?? throw new InvalidOperationException($"Could not determine the CPU architecture from the PE image at '{path}'. The image may be corrupt or of an unsupported format.");

---

Task 4: Fix typo in PlatformResources.resx

Priority: Low
Estimated Effort: Small

In src/Platform/Microsoft.Testing.Platform/Resources/PlatformResources.resx, the string for CompositeServiceFactoryInstanceAlreadyRegistered contains a typo:

<!-- Current (line 127): -->
<value>The same instance of 'CompositeExtensonFactory' is already registered</value>
<!-- Fixed: -->
<value>The same instance of 'CompositeExtensionFactory' is already registered</value>

Note: After fixing the .resx file, rebuild the project to regenerate the .xlf localization files. Do not manually edit .xlf files.

---

Task 5: Improve translator comment coverage in resource files

Priority: Medium
Estimated Effort: Large

Resource file translator comment coverage is critically low across the board:

File	Coverage
MSTest.Analyzers/Resources.resx	5/171 (2.9%)
FrameworkMessages.resx	7/95 (7.4%)
Resource.resx (adapter)	10/108 (9.3%)
PlatformResources.resx	19/199 (9.5%)

Each <data> entry that represents a user-visible string should have a <comment> element describing: what the string is shown for, what the placeholder arguments mean (if any {0}, {1}, etc. are present), and any length or formatting constraints. For example:

<data name="AreEqualFailMsg" xml:space="preserve">
  <value>Expected:<{1}>. Actual:<{2}>. {0}</value>
  <comment>Shown when Assert.AreEqual fails. {0} = optional user message, {1} = expected value, {2} = actual value.</comment>
</data>

Start with the highest-traffic files: FrameworkMessages.resx (assertion messages seen by all users) and PlatformResources.resx (platform error messages).

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type
2026-05-08	Diagnostic & Error Message Quality	Custom

---

🎯 Recommendations

Immediate Actions (This Week)

1. Fix the "err" placeholder in JsonObjectSerializer.cs — Priority: High (blocks debugging)
2. Fix the CompositeExtensonFactory typo in PlatformResources.resx — Priority: Low (quick win)

Short-term Actions (This Month)

1. Add [CallerArgumentExpression] to StringAssert — Priority: High (improves test failure messages)
2. Add [CallerArgumentExpression] to CollectionAssert — Priority: High (improves test failure messages)
3. Improve translator comment coverage in FrameworkMessages.resx and PlatformResources.resx — Priority: Medium

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-05-09 — Focus area selected based on diversity algorithm

Generated by Repository Quality Improver · ● 967.2K · ◷

• expires on May 10, 2026, 5:54 AM UTC

[github-actions]

This issue was automatically closed because it expired on 2026-05-10T05:54:59.784Z.

Closed by Workflow