Fix xUnit Theory serialization warnings #3186

paulmedynski · 2025-02-28T14:21:39Z

xUnit makes 2 passes over unit tests:

Discover all of the tests, and for Theory tests, enumerate all of the test data.
Run all of the tests, enumerating the test data again.

If the enumeration won't produce the same data in both steps, xUnit emits a warning. We have many test cases whose Theory data doesn't enumerate identically in both passes, for a variety of reasons. These tests will still run correctly, but they won't appear in test runner UIs like Visual Studio. Instead, xUnit collapses them into a single test for these UIs.

The warnings are clutter in the console (where I like to run tests), so xUnit provides a mechanism to disable enumeration completely in the discovery phase. I have updated the offending tests to use the MemberData DisableDiscoveryEnumeration flag. This doesn't affect how the tests run, and it doesn't change how UIs display them (collapsed). All it does is suppress the console warnings.

Further reading here: https://xunit.net/faq/theory-data-stability-in-vs

I also added 4 new targets to build.proj to make running the tests easier. The existing test running targets function as before.

Commit notes:

Suppressed xUnit discovery data enumeration warnings in Function and Manual test projects.
Fixed xUnit test data enumeration warnings specific to .NET Framework (enums in the GAC).
Added targets to run Windows and Unix tests.
Cleaned up test run command lines for improved maintenance and runtime logging.

paulmedynski · 2025-02-28T14:29:58Z

build.proj

+  <Target Name="RunFunctionalTestsWindows" Condition="'$(IsEnabledWindows)' == 'true'">
+    <PropertyGroup>
+      <TestCommand>
+        $(DotnetPath)dotnet test "@(FunctionalTestsProj)"


Formatted for readability, and then squished back into a single line since <Exec> treats each line in the Command as a separate command!

paulmedynski · 2025-02-28T14:31:12Z

build.proj

+        --filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
+        --logger:"trx;LogFilePrefix=Functional-Windows$(TargetGroup)-$(TestSet)"
+      </TestCommand>
+      <TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>


Modern dotnet tooling supports String.ReplaceLineEndings(), but we're using the old .NET Framework msbuild.exe tooling to build, so we can't have the nice toys :(

paulmedynski · 2025-02-28T14:33:09Z

...ts/FunctionalTests/AlwaysEncryptedTests/SqlColumnEncryptionCertificateStoreProviderShould.cs

+        [MemberData(
+            nameof(CEKEncryptionReversalData)
+#if NETFRAMEWORK
+            // .NET Framework puts system enums in something called the Global


I've left a comment once in each file where we're using DisableDiscoveryEnumeration, to make it clear why we're using that flag.

paulmedynski · 2025-02-28T14:34:31Z

...ts/FunctionalTests/AlwaysEncryptedTests/SqlColumnEncryptionCertificateStoreProviderShould.cs

@@ -505,33 +519,25 @@ public override IEnumerable<Object[]> GetData(MethodInfo testMethod)
            }
        }

-        public class CEKEncryptionReversalParameters : DataAttribute
+        public static IEnumerable<object[]> CEKEncryptionReversalData()


Since xUnit only supports DisableDiscoveryEnumeration on MemberData attributes, I have converted all of the data sources to be compatible with MemberData.

You will see similar changes throughout the tests, where data source classes become methods.

paulmedynski · 2025-02-28T14:35:29Z

...ts/FunctionalTests/AlwaysEncryptedTests/SqlColumnEncryptionCertificateStoreProviderShould.cs

-        [CEKEncryptionReversalParameters]
+        [MemberData(
+            nameof(CEKEncryptionReversalData)
+#if NETFRAMEWORK


.NET doesn't have a GAC, so these tests will enumerate correctly for those targets. We only need to disable for .NET Framework targets.

paulmedynski · 2025-02-28T14:35:49Z

src/Microsoft.Data.SqlClient/tests/FunctionalTests/AmbientTransactionFailureTest.cs

@@ -4,6 +4,7 @@

 using System;
 using System.Runtime.InteropServices;
+using System.Threading;


I don't think this is necessary.

paulmedynski · 2025-02-28T14:37:53Z

src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlMetaDataTest.cs

+#endif
+            )]
+        [MemberData(
+            nameof(ConstructorTextData)


This adds Text to the mix, which seems to have been missing from the InlineData.

paulmedynski · 2025-02-28T14:41:21Z

src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ConversionTests.cs

@@ -565,7 +584,20 @@ private void ExecuteAndCheckForError(SqlCommand sqlCmd, bool expectError)
                try
                {
                    sqlCmd.ExecuteNonQuery();
-                    Assert.Fail("We should have gotten an error but passed instead.");
+                    StringBuilder builder = new(


With broken (now disabled) enumeration, xUnit doesn't include the test data in any failure output, so we need to do it ourselves.

This test fails for me on main as well, so I wanted to see what data set was causing the problem. Turns out to be the Date data set (line 1450).

paulmedynski · 2025-02-28T14:45:47Z

...rosoft.Data.SqlClient/tests/tools/Microsoft.Data.SqlClient.TestUtilities/config.default.json

@@ -1,6 +1,13 @@
 {
+    // Typical Windows SQL Server on the local host via TCP.


These comments are breaking our CI "Update config.json" task because PowerShell's ConvertFrom-Json fails But as of PowerShell v6, ConvertFrom-Json handles comments:

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertfrom-json?view=powershell-7.5

Are we really using ancient PowerShell 5.1 from 2016? I will have to dig into this further.

- Suppressed xUnit discovery data enumeration warnings in Function and Manual test projects. - Fixed xUnit test data enumeration warnings specific to .NET Framework (enums in the GAC). - Added targets to run Windows and Unix tests. - Cleaned up test run command lines for improved maintenance and runtime logging.

- Added installation of PowerShell Core (pwsh) as a dotnet global tool so we can use the ADO task PowerShell@2 in pwsh mode on all agents/images.

- Removed attempt to install PowerShell Core. Adding it to the 1ES images instead.

- Trying pwsh (PowerShell Core 7.x) again for config file update job.

- Updating pipeline to use my new Win11 1ES image with pwsh.

codecov · 2025-03-05T18:54:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.52%. Comparing base (b875e1d) to head (e928eb1).
Report is 21 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3186      +/-   ##
==========================================
- Coverage   72.79%   72.52%   -0.27%     
==========================================
  Files         285      289       +4     
  Lines       59156    59543     +387     
==========================================
+ Hits        43060    43183     +123     
- Misses      16096    16360     +264

Flag	Coverage Δ
addons	`92.58% <ø> (ø)`
netcore	`75.08% <ø> (-0.38%)`	⬇️
netfx	`71.15% <ø> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Reverting pipeline to use the newly rebuilt ADO-CI-Win11 image.

Copilot

Pull Request Overview

This PR addresses xUnit Theory serialization warnings by replacing InlineData and ClassData attributes with MemberData attributes that disable discovery enumeration on .NET Framework. It also refactors several test data providers across multiple test files and updates pipeline configuration steps to use pwsh consistently.

Migrate test data attributes from InlineData/ClassData to MemberData with DisableDiscoveryEnumeration for improved xUnit behavior.
Refactor test data providers into static methods returning IEnumerable<object[]>.
Update pipeline YAML steps to use "pwsh" instead of "powershell" for configuration consistency.

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlParameterTest.cs	Refactored xUnit test attributes for Constructor3, Constructor6, and Constructor7 tests using MemberData.
src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ConversionTests.cs	Replaced ClassData with MemberData in multiple tests to suppress discovery warnings and enhanced error logging details.
src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlCommandSetTest.cs	Introduced MemberData for CommandType tests with appropriate disable flags on .NET Framework.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionTestWithSSLCert/CertificateTestWithTdsServer.cs	Updated MemberData usage to disable discovery enumeration for connection tests on different platforms.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/JsonTest/JsonBulkCopyTest.cs	Converted JsonBulkCopyTestData from a class to a static method and updated test attributes accordingly.
src/Microsoft.Data.SqlClient/tests/FunctionalTests/AlwaysEncryptedTests/SqlColumnEncryptionCertificateStoreProviderShould.cs	Replaced custom data attribute with MemberData for CEK encryption reversal and valid certificate paths tests.
src/Microsoft.Data.SqlClient/tests/FunctionalTests/AmbientTransactionFailureTest.cs	Updated MemberData for exception test data to disable discovery enumeration.
src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ApiShould.cs	Refactored multiple tests to use MemberData for connection string providers with disable discovery flags.
src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlClientFactoryTest.cs	Updated FactoryMethodTest to use MemberData with disable discovery enumeration for improved stability.
src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/TestFixtures/DatabaseHelper.cs	Consolidated AE connection string providers into a static DataHelpers class with static test data methods.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/DataReaderTest/DataReaderStreamsTest.cs	Updated MemberData attributes for various DataReader stream tests with disable discovery flags on .NET Framework.
src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlDataRecordTest.cs	Refactored test data providers for UDT and XXX value tests from ClassData to MemberData.
eng/pipelines/common/templates/steps/update-config-file-step.yml	Replaced "powershell:" with "pwsh:" in the YAML steps for updating the config file.
src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlMetaDataTest.cs	Refactored several tests to use MemberData with DisableDiscoveryEnumeration, and added new static test data providers for constructor tests.

Comments suppressed due to low confidence (2)

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/JsonTest/JsonBulkCopyTest.cs:27

The parameter 'enableStraming' appears to be a typo. It is recommended to rename it to 'enableStreaming' for clarity and consistency.

public void TestJsonBulkCopy(CommandBehavior cb, bool enableStraming, int jsonArrayElements, int rows)

eng/pipelines/common/templates/steps/update-config-file-step.yml:129

[nitpick] The task key has been updated from 'powershell:' to 'pwsh:'. Please ensure that all related pipeline configuration files consistently use 'pwsh' as the identifier for PowerShell steps.

- pwsh: |

mdaigle · 2025-03-18T15:53:58Z

src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlClientFactoryTest.cs

+            // xUnit can't consistently serialize the data for this test, so we
+            // disable enumeration of the test data to avoid warnings on the
+            // console.
+            DisableDiscoveryEnumeration = true)]


Would a custom serializer help in this case? Not to say we should do it in this PR, just curious if you experimented with it.

https://xunit.net/docs/getting-started/v3/custom-serialization

Yes looked into that, but didn't want to figure out serialization for all of the affected tests as part of this PR. The changes here don't affect how the tests run, and they don't change how they're visualized in IDEs. They just suppress the warnings on the console. I've opened an ADO task to track fixing the serialization itself.

paulmedynski commented Feb 28, 2025

View reviewed changes

paulmedynski force-pushed the dev/paul/fix-test-theory-data-warnings branch from a04396b to ae09fcc Compare March 3, 2025 14:19

paulmedynski added 5 commits March 3, 2025 10:25

User Story 33279: Fix xUnit Theory serialization warnings

44b29be

- Added installation of PowerShell Core (pwsh) as a dotnet global tool so we can use the ADO task PowerShell@2 in pwsh mode on all agents/images.

User Story 33279: Fix xUnit Theory serialization warnings

034522f

- Added installation of PowerShell Core (pwsh) as a dotnet global tool so we can use the ADO task PowerShell@2 in pwsh mode on all agents/images.

User Story 33279: Fix xUnit Theory serialization warnings

430778e

- Removed attempt to install PowerShell Core. Adding it to the 1ES images instead.

User Story 33279: Fix xUnit Theory serialization warnings

5c5fa2e

- Trying pwsh (PowerShell Core 7.x) again for config file update job.

User Story 33279: Fix xUnit Theory serialization warnings

334959b

- Updating pipeline to use my new Win11 1ES image with pwsh.

User Story 33279: Fix xUnit Theory serialization warnings

e928eb1

- Reverting pipeline to use the newly rebuilt ADO-CI-Win11 image.

paulmedynski marked this pull request as ready for review March 14, 2025 14:33

paulmedynski requested review from Copilot and a team March 14, 2025 14:33

paulmedynski added this to the 7.0-preview1 milestone Mar 14, 2025

Copilot AI reviewed Mar 14, 2025

View reviewed changes

cheenamalhotra approved these changes Mar 18, 2025

View reviewed changes

cheenamalhotra added Area\Tests Issues that are targeted to tests or test projects Area\Engineering Use this for issues that are targeted for changes in the 'eng' folder or build systems. labels Mar 18, 2025

mdaigle reviewed Mar 18, 2025

View reviewed changes

paulmedynski requested a review from mdaigle March 18, 2025 18:51

mdaigle approved these changes Mar 18, 2025

View reviewed changes

paulmedynski merged commit f3ae72a into main Mar 18, 2025
252 checks passed

paulmedynski deleted the dev/paul/fix-test-theory-data-warnings branch March 18, 2025 19:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix xUnit Theory serialization warnings #3186

Fix xUnit Theory serialization warnings #3186

Uh oh!

paulmedynski commented Feb 28, 2025 •

edited

Loading

Uh oh!

paulmedynski Feb 28, 2025 •

edited

Loading

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

paulmedynski Feb 28, 2025 •

edited

Loading

Uh oh!

paulmedynski Feb 28, 2025

Uh oh!

codecov bot commented Mar 5, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

mdaigle Mar 18, 2025

Uh oh!

mdaigle Mar 18, 2025

Uh oh!

paulmedynski Mar 18, 2025

Uh oh!

Uh oh!

Uh oh!

		@@ -1,6 +1,13 @@
		{
		// Typical Windows SQL Server on the local host via TCP.

Fix xUnit Theory serialization warnings #3186

Fix xUnit Theory serialization warnings #3186

Uh oh!

Conversation

paulmedynski commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paulmedynski Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulmedynski Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

paulmedynski commented Feb 28, 2025 •

edited

Loading

paulmedynski Feb 28, 2025 •

edited

Loading

paulmedynski Feb 28, 2025 •

edited

Loading

codecov bot commented Mar 5, 2025 •

edited

Loading