Add test dependencies: [DependsOn] attribute and testconfig.json declarations - #10260
Conversation
Introduce test dependencies to MSTest: a test can declare that it runs after one or more other tests, either with the new [DependsOn] attribute or through an XML dependency chain file referenced from runsettings/testconfig. The declarations form a directed acyclic graph rather than a flat order, which is the point of the design: tests that share a prerequisite become runnable at the same moment and the in-assembly scheduler runs them concurrently. The flat ordering alternative would serialize them. - Skip, don't fail: a test whose prerequisite did not pass is reported as skipped (transitively), so one root cause reads as one failure surrounded by labelled skips. ProceedOnFailure is the escape hatch for audit/cleanup tests. - Cycles are detected before anything runs; the tests in the cycle fail with the cycle path and the rest of the run proceeds. - Test-level edges are projected onto scheduling chunks (class- or method-level). A cycle that exists only in the class-level projection is reported with advice to use MethodLevel instead of deadlocking. - A parallelizable test depending on a [DoNotParallelize] test is demoted into the sequential phase, transitively, since that phase runs last. - A dependency matching no test in the run warns and is ignored, so --filter and running a single test from an IDE keep working. TestDependencyGraph.Build returns null when no test declares a dependency, so runs that do not use the feature stay on exactly the existing code path. Covered by 18 graph unit tests, 8 attribute unit tests, and acceptance tests that assert both the ordering and the overlap of independent branches from recorded timings, across net462/net8.0/net10.0. See RFC 022. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
Replace the separate XML dependency chain file with declarations inline in testconfig.json under mstest:execution:dependencies. testconfig.json is supplied only on the Microsoft.Testing.Platform path (the VSTest entry points pass configuration: null), so configured dependencies are now an MTP capability. A separate RunSettings-reachable file format was the alternative; it would have meant a second syntax, parser and set of diagnostics for the same concept in order to serve a host that is being superseded. The [DependsOn] attribute still works on both hosts. The section supports 'chains' (each entry waits for the previous one) and 'nodes' (one test naming several prerequisites, with proceedOnFailure), which together cover the flat and tree cases the XML format covered. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
The asset's chain is First -> Second -> Third with Verify and Audit hanging off Second, and Second fails. Third is therefore skipped too - it is downstream of Second in the chain - so the run is 1 failed, 2 skipped, 2 succeeded, not 1 skipped. Assert the skip reason and that neither skipped test executed its body. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
1. Deadlock when a chunk throws. The worker consumed a permit and dequeued a chunk, then an exception from the chunk skipped both the dependent release and the completion count - so dependents never became ready, the shutdown permits were never issued, and every other worker blocked forever on WaitAsync while Task.WhenAll waited for them. One faulty chunk hung the whole run, and the error handler written for exactly that case was unreachable. The bookkeeping now runs in a finally. Regression test asserts the run completes rather than hangs (it deadlocks without the fix). 2. A cycle that existed only in the class-level projection dropped every chunk edge, leaving the affected tests unordered. That was worse than losing the parallelism: the run-time gate cannot distinguish 'has not run yet' from 'did not pass', so dependents were skipped nondeterministically - varying with worker count and thread timing - while the run still reported success. Those tests are now moved into the sequential phase, where the topological order is honoured exactly; only their parallelism is lost, and the error says how to get it back. 3. A class-level [DependsOn(nameof(Setup))] is expanded onto every method of the class, including Setup, which made Setup depend on itself. The graph reported a cycle the user never wrote, failed Setup, and cascaded a skip over the whole class. That generated self-edge is now dropped at discovery; a self reference written directly on a method is still kept, since there the user really did name the test they were annotating. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
- The class-level projection notice moves from Errors to Warnings. It describes
a recovered downgrade - the declared order is still honoured - so reporting it
at error severity was a poor signal, and because every entry in Errors is
joined into the failure message stamped on tests caught in a *real* cycle, it
also contaminated unrelated failures.
- RFC 022 still documented the pre-fix behaviour ('the projected chunk edges are
dropped'); rewrite the section for demotion, restate the deadlock-freedom
argument in terms of it, and correct the now-answered unresolved question.
- Purge 'dependency chain file' from public XML docs and test constants; the
format has been testconfig.json since 91c6b9e.
- Correct the [DoNotParallelize] rationale on the acceptance class: the assets
are not shared between its test methods. The attribute is still right, for the
real reason - the overlap assertions need a quiet machine.
- TryParse follows bool + out; resx formatting; RFC test count.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
There was a problem hiding this comment.
Pull request overview
Adds DAG-based MSTest dependencies through [DependsOn] and MTP testconfig.json declarations.
Changes:
- Adds dependency APIs, configuration parsing, metadata transport, and diagnostics.
- Introduces graph scheduling, failure propagation, cycle handling, and sequential demotion.
- Adds API manifests, localization resources, documentation, and extensive tests.
Show a summary per file
| File | Description |
|---|---|
src/TestFramework/TestFramework/Attributes/Lifecycle/DependsOnAttribute.cs |
Defines the public attribute. |
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt |
Tracks the new public API. |
src/TestFramework/TestFramework/Resources/FrameworkMessages.resx |
Adds attribute validation text. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.cs.xlf |
Adds Czech localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.de.xlf |
Adds German localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf |
Adds Spanish localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.fr.xlf |
Adds French localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.it.xlf |
Adds Italian localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ja.xlf |
Adds Japanese localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ko.xlf |
Adds Korean localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pl.xlf |
Adds Polish localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf |
Adds Portuguese localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.ru.xlf |
Adds Russian localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.tr.xlf |
Adds Turkish localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf |
Adds Simplified Chinese localization source. |
src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hant.xlf |
Adds Traditional Chinese localization source. |
src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/TestDependencyInfo.cs |
Models and encodes dependency edges. |
src/Adapter/MSTestAdapter.PlatformServices/ObjectModel/UnitTestElement.cs |
Carries discovered dependencies. |
src/Adapter/MSTestAdapter.PlatformServices/Discovery/TypeEnumerator.cs |
Reads and merges dependency attributes. |
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyDeclaration.cs |
Applies configuration declarations. |
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyGraph.cs |
Resolves and schedules the DAG. |
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyCoordinator.cs |
Tracks prerequisite outcomes. |
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs |
Gates and skips dependent tests. |
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Parallelization.cs |
Executes dependency-aware chunks. |
src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.cs |
Stores configured dependencies. |
src/Adapter/MSTestAdapter.PlatformServices/MSTestSettings.Configuration.cs |
Parses dependency configuration. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/Resource.resx |
Adds scheduler diagnostics. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.cs.xlf |
Adds Czech diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.de.xlf |
Adds German diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.es.xlf |
Adds Spanish diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.fr.xlf |
Adds French diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.it.xlf |
Adds Italian diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ja.xlf |
Adds Japanese diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ko.xlf |
Adds Korean diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pl.xlf |
Adds Polish diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.pt-BR.xlf |
Adds Portuguese diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.ru.xlf |
Adds Russian diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.tr.xlf |
Adds Turkish diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hans.xlf |
Adds Simplified Chinese diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/Resources/xlf/Resource.zh-Hant.xlf |
Adds Traditional Chinese diagnostic sources. |
src/Adapter/MSTestAdapter.PlatformServices/InternalAPI/InternalAPI.Unshipped.txt |
Tracks new internal APIs. |
src/Adapter/MSTest.TestAdapter/AdapterTestProperties.cs |
Registers dependency transport metadata. |
src/Adapter/MSTest.TestAdapter/Extensions/UnitTestElementExtensions.cs |
Serializes dependencies to test cases. |
src/Adapter/MSTest.TestAdapter/Extensions/TestCaseExtensions.cs |
Restores dependencies during execution. |
src/Adapter/MSTest.TestAdapter/InternalAPI/InternalAPI.Unshipped.txt |
Tracks adapter internal APIs. |
src/Adapter/MSTest.TestAdapter/InternalAPI/uwp/InternalAPI.Unshipped.txt |
Tracks UWP transport API. |
test/UnitTests/TestFramework.UnitTests/Attributes/DependsOnAttributeTests.cs |
Tests the public attribute contract. |
test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Discovery/TypeEnumeratorTests.cs |
Tests class-level self-edge handling. |
test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestDependencyGraphTests.cs |
Tests graph resolution and ordering. |
test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/Execution/TestExecutionManagerTests.cs |
Tests scheduler exception handling. |
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs |
Exercises dependencies end to end. |
docs/testconfig.schema.json |
Documents configuration structure. |
docs/RFCs/022-Test-Dependencies.md |
Specifies the feature design. |
docs/Changelog.md |
Announces the feature. |
Review details
- Files reviewed: 54/54 changed files
- Comments generated: 3
- Review effort level: Medium
- Merging the same prerequisite declared twice kept the *permissive* ProceedOnFailure, so a class-level opt-out silently overrode a method-level default and ran a test whose precondition had failed. That contradicted the rule already applied across distinct prerequisites (ResolveEdges) and stated in the attribute docs and RFC. Merge conservatively; regression test covers the class-plus-method case. - An empty chain stores nothing under its index, and IConfiguration's indexer cannot tell an absent key from a null one, so a stray [] looked like the end of the outer array and silently discarded every chain after it. The schema now requires at least two entries per chain, and the runtime warns and continues past a single empty element instead of truncating. - The attribute's own XML docs still told consumers a class-level projection cycle fails the run; it has demoted to the sequential phase since e913940. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
This comment has been minimized.
This comment has been minimized.
Two remaining conflations in the dependency diagnostics, both making a message claim more than it should: - Every broken test was failed with string.Join(graph.Errors), so an assembly containing two unrelated cycles stamped both cycle paths onto every failure. FindCycles now attributes each cycle's description to that cycle's own members and BrokenTests carries it per test, so a failure names the cycle the test is actually in. This is the same conflation already removed for the projection warning - it was only fixed on one side. - Kahn's algorithm cannot settle a chunk that is on a cycle *or* merely downstream of one, and the class-level warning named the whole unsettled set as classes that 'depend on each other' - false for the downstream ones. HasChunkCycle now reports the two separately: everything blocked is still demoted out of the parallel phase, but only the chunks genuinely in the cycle are named. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
This comment has been minimized.
This comment has been minimized.
The test asserted only that no diagnostic was produced, which would still hold if the graph came back with the wrong chunks - its name claims the declarations schedule cleanly under MethodLevel, so it should verify that. Now asserts one chunk per test, an empty sequential phase, and the actual chunk edges. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (2)
src/Adapter/MSTestAdapter.PlatformServices/InternalAPI/InternalAPI.Unshipped.txt:36
- This baseline still describes the old
BrokenTestsreturn type and omits the newBrokenTestnested API andMSTestSettings.DeclaredDependencies. PlatformServices tracks internal members (for example,InternalAPI.Shipped.txt:326-346tracksMSTestSettings), so the API analyzers will report both stale and undeclared APIs. Update the baseline to match the current source.
Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestDependencyGraph.BrokenTests.get -> System.Collections.Generic.IReadOnlyList<Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement!>!
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyGraph.cs:675
- Removing blocked vertices with no blocked dependents does not isolate cycle members when two cycles are connected. For class edges
A↔B,C→B,D→C, andD↔E, every vertex has a blocked dependent, soCsurvives even though it is only a bridge and cannot reach itself; the warning incorrectly names it as cyclic. Compute actual strongly connected components for the diagnostic while continuing to demote all blocked chunks.
/// <summary>
/// Narrows the blocked chunks down to those actually on a cycle, by repeatedly discarding any chunk that
/// no other blocked chunk depends on. Such a chunk can reach the cycle but nothing returns to it, so it is
/// downstream of the cycle rather than part of it; what survives is exactly the chunks that can reach
/// themselves.
- Files reviewed: 54/54 changed files
- Comments generated: 3
- Review effort level: Medium
This comment has been minimized.
This comment has been minimized.
…ransport - A configured dependent of 'Ns.Class.*' expands onto every test of the class, including the prerequisite when it lives there too, so 'Ns.Class.* dependsOn Ns.Class.Setup' made Setup depend on itself - reported as a cycle, failing Setup and skipping the class. Discovery already dropped this generated self-edge for a class-level [DependsOn]; the configuration path did not. - InternalAPI.Unshipped.txt still declared BrokenTests as returning IReadOnlyList<UnitTestElement> and omitted the nested BrokenTest type. This does not currently fail the build (verified), but the file exists to be an accurate record of the internal surface. - Nothing exercised the VSTest TestProperty transport for dependencies: the acceptance assets run on the native MTP path, so a registration or conversion regression would silently disable [DependsOn] under VSTest. The new round-trip test clears LocalExtensionData first, because ToTestCase caches the element there and returning that cached instance would make the test pass without the encoded property being read at all - which is what the equivalent ResourceLock tests do today. Verified it fails when the decode is removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
…keeping ClassCleanupManager's countdown is built from every *selected* test, and end-of-assembly cleanup is gated on every class reaching zero (ShouldRunEndOfAssemblyCleanup => _remainingTestCountsByClass.IsEmpty). Tests whose outcome this feature decides without running them - skipped because a prerequisite did not pass, or failed because they are in a cycle - never reached UnitTestRunner, so they never decremented that countdown. The class therefore never completed: its [ClassCleanup] was silently lost, and with it the whole assembly's [AssemblyCleanup]. Verified: a class whose dependent is skipped ended the run with ClassCleanupCount == 0, even though the prerequisite had run and so [ClassInitialize] had executed. This is the same situation the [TestFilterProvider] feature already solved for dropped tests - selected, counted, never run - so rather than grow a second implementation, both paths now call the existing bookkeeping through a new UnitTestRunner.NotifyTestNotRunAsync entry point. FinishFilteredOutTestAsync is renamed FinishTestThatDidNotRunAsync to match its now-shared purpose. Any results produced by cleanup that these calls trigger (only non-empty when a cleanup fails) are appended to the test's reported results, mirroring the filter path. Both regression tests were verified to fail without the fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (3)
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs:15
- This summary still says the recorded timestamps make branch overlap checkable, but overlap is now deliberately verified by the bounded rendezvous; timestamps only verify prerequisite ordering. Align the summary with the non-flaky mechanism described immediately below.
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs:212 - When a dependency-skipped test triggers a failing class/assembly cleanup,
cleanupResultscarriesAssociatedUnitTestElementpointing to the last runnable test, but the native MTP recorder ignores that property and converts every result using thetestargument passed here. The cleanup failure is therefore published against the skipped dependent (potentially replacing its skipped outcome) instead of the test selected by the cleanup bookkeeping. Dispatch cleanup results using their associated element or teach the native recorder to honor the association, as the VSTest converter already does.
await SendTestResultsAsync(
test,
[TestTools.UnitTesting.TestResult.CreateIgnoredResult(reason), .. cleanupResults],
now,
now,
_testResultRecorder).ConfigureAwait(false);
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestDependencyGraph.cs:309
- A valid whole-class dependency on a class containing only the dependent reaches this branch after its sole self-match is intentionally removed.
matchesis nonempty, butaddedis zero, so the runtime incorrectly warns that the target “matches no test.” Reserve this diagnostic for a genuinely empty lookup; a self-only whole-class dependency is simply a no-op.
if (added == 0)
{
warnings.Add(string.Format(
CultureInfo.CurrentCulture,
Resource.DependsOnTargetNotFound,
tests[i].TestMethod.FullyQualifiedName,
dependency.DescribeTarget()));
continue;
}
- Files reviewed: 59/59 changed files
- Comments generated: 0 new
- Review effort level: Medium
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
ProceedOnFailure_CanBeSet asserted only the true case, so a property that ignored its setter and always returned true would have passed. Assert the default first. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a6be10-1c72-4887-b7a3-8a93e19805f0
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (2)
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Parallelization.cs:183
- This pre-sort runs even when
TestDependencyGraph.Buildwill returnnull, so enablingOrderTestsByNameInClassnow changes ordinary parallel runs that do not use dependencies. In particular, MethodLevel chunks are globally queued by type/method name here, whereas previously the setting sorted only within each chunk inExecuteTestsWithTestRunnerAsync; this breaks the stated no-dependency fast-path invariant. Apply configured edges first, detect whether a graph is needed, and only pre-sort the graph input.
if (MSTestSettings.CurrentSettings.OrderTestsByNameInClass && !MSTestSettings.CurrentSettings.RandomizeTestOrder)
{
testsToRun =
[
.. testsToRun
.OrderBy(t => t.TestMethod.HasManagedMethodAndTypeProperties ? t.TestMethod.ManagedTypeName : null, StringComparer.Ordinal)
.ThenBy(t => t.TestMethod.HasManagedMethodAndTypeProperties ? t.TestMethod.ManagedMethodName : null, StringComparer.Ordinal),
];
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs:212
- When this notification triggers a failing class/assembly cleanup, the returned result has
AssociatedUnitTestElementset to the last runnable test, but this call submits every result using the skipped dependent. VSTest corrects that association inTestResultExtensions, whileMtpTestResultRecorderignores it and builds the node from the supplied element, so native MTP reports both the skip and cleanup failure against the skipped test instead of the test that owns the cleanup result. Make the MTP recording path honorAssociatedUnitTestElement(and cover this dependency-skip case).
await SendTestResultsAsync(
test,
[TestTools.UnitTesting.TestResult.CreateIgnoredResult(reason), .. cleanupResults],
now,
now,
_testResultRecorder).ConfigureAwait(false);
- Files reviewed: 59/59 changed files
- Comments generated: 0 new
- Review effort level: Medium
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Correcting one point in the automated analysis above, since it contradicts the earlier build-failure comment and could lead to the wrong fix landing here:
It hasn't. This branch touches zero files under
All three failures are the same The specific drift, for whoever picks this up: So the fix is |
There was a problem hiding this comment.
Review details
Comments suppressed due to low confidence (2)
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDependencyExecutionTests.cs:15
- This summary is stale: timestamps now verify ordering only; branch overlap is established by the bounded rendezvous described below. Saying the recorded milliseconds make both claims checkable contradicts the test and could encourage reintroducing the timing-based overlap assertion that was removed for flakiness.
src/TestFramework/TestFramework/Attributes/Lifecycle/DependsOnAttribute.cs:200 - This rationale incorrectly describes
ProceedOnFailureas opting out of waiting, although ordering is always enforced as stated above. It only opts the dependent into running after a prerequisite has completed unsuccessfully; please phrase it in terms of failure handling so the public API documentation does not contradict itself.
- Files reviewed: 59/59 changed files
- Comments generated: 0 new
- Review effort level: Medium
This comment has been minimized.
This comment has been minimized.
|
Correcting myself on the mechanism, and adding a caution for whoever fixes My earlier claim about ordering was wrong. I said the resx and What <trans-unit id="GlobalTestFixtureShouldBeValidDescription">
- <target state="translated">Les méthodes marquées par « [GlobalTestInitialize] » ou « [GlobalTestCleanup] » …
+ <target state="new">Methods marked with '[GlobalTestInitialize]' or '[GlobalTestCleanup]' should follow …It discards the existing French translation, reverting Flagging it because the fix belongs on Two other updates:
|
🔴 Build Failure AnalysisAll 6 build legs (Linux/macOS/Windows × Release/Debug) fail with the same error: Root cause
FixRegenerate the XLF file by running: dotnet msbuild src/Analyzers/MSTest.Analyzers/MSTest.Analyzers.csproj /t:UpdateXlfThen commit
|
…rdering-dependencies
🧪 Test quality grade — PR #10260No new or modified test methods were identified in the changed regions Re-run with
|
Adds test dependencies to MSTest: a test can declare that it runs after one or more other tests, either with a new
[DependsOn]attribute or throughtestconfig.json.The declarations form a directed acyclic graph, not a flat list. That is the point of the design: when several tests share a prerequisite they all become runnable at the same moment, and the in-assembly scheduler runs them concurrently. A flat ordering attribute would serialize them.
Implements RFC 022. Addresses long-standing requests #25 and #3162.
API
Overloads cover a method of the same class, every test of another class (
typeof(C)), and a specific method of another class (typeof(C), nameof(C.M)). Repeating the attribute is how a test declares several prerequisites — consistent with every otherAllowMultipleattribute in MSTest.The same graph can be declared outside the test source, for orchestrating tests somebody else owns or keeping a pipeline visible in one reviewable place:
{ "mstest": { "execution": { "dependencies": { "chains": [ [ "Contoso.Tests.SetupTests.CreateDatabase", "Contoso.Tests.ImportTests.ImportCatalog" ] ], "nodes": [ { "test": "Contoso.Tests.ReportTests.WriteAudit", "dependsOn": [ "Contoso.Tests.CheckoutTests.PlaceOrder", "Contoso.Tests.ImportTests.*" ], "proceedOnFailure": true } ] } } } }chainsis the flat case,nodesthe tree case. This part is Microsoft.Testing.Platform only, deliberately:testconfig.jsonis not supplied to the VSTest entry points, which passconfiguration: null. A separate RunSettings-reachable file format was prototyped and dropped — it meant a second syntax, parser and set of diagnostics for the same concept, to serve a host that is being superseded. The attribute works on both hosts.Semantics
ProceedOnFailure--filterand running one test from an IDE keep working.[DoNotParallelize]prerequisiteTestDependencyGraph.Buildreturnsnullwhen no test declares a dependency, so runs that don't use the feature stay on exactly the existing code path.Review findings fixed in this PR
Two review rounds (correctness, then repo conventions/API) found four issues, all fixed here:
Deadlock (High). An exception in a chunk skipped both the dependent-release and the completion count, so every other worker blocked forever on a semaphore permit that would never be issued and
Task.WhenAllnever returned — one faulty chunk hung the whole run, and the error handler written for exactly that case was unreachable. Bookkeeping now runs in afinally. The regression test was verified to genuinely hang against the unfixed code.Nondeterministic silent test loss (High). A cycle that existed only in the class-level projection dropped every chunk edge, on the assumption that the run-time gate would still enforce order. It can't: the gate cannot distinguish "hasn't run yet" from "didn't pass", so dependents were skipped nondeterministically — varying with worker count and thread timing — while the run still exited 0. Those tests are now demoted to the sequential phase, where the topological order holds exactly; only their parallelism is lost, and only for the classes involved.
Class-level self-cycle (Medium).
[DependsOn(nameof(Setup))]applied to a class expanded ontoSetupitself, failing it as a cycle and cascading a skip over the whole class. The generated self-edge is dropped at discovery; a self-reference written on a method is still kept, since there the user really did name the test they were annotating.Message severity/conflation (Moderate). The class-level projection notice described a recovered downgrade but was reported as an error, and — because every
Errorsentry is joined into the message stamped on tests in a real cycle — it contaminated unrelated failures. It is now a warning.Testing
ProceedOnFailuremerging, ordering determinism, encode/decode) + 8 attribute.Full
build.cmd -c Release -packclean. 5049 + 5810 + 280 existing unit tests pass.Notes for reviewers
WellKnownTypeNamesregistration, usage telemetry, and NativeAOT source-generation support — all of which[ResourceLock]shares to varying degrees.MSTest.TestAdapter/InternalAPI/InternalAPI.Unshipped.txt; that is hygiene, unrelated to the feature, and can be ignored while reviewing.