Add dotnet test --list-devices support for MAUI

[jonathanpeppers]

Expands on #54295 (which added dotnet test --device) to implement the --list-devices switch on dotnet test, as described in dotnet-run-for-maui.md. This lets users discover the device identifiers they can pass to --device for MAUI/Android/iOS test projects without first having to run a build.

Behavior

dotnet test --list-devices mirrors dotnet run --list-devices:

• Resolves the project (rejects solutions, since each project may have its own device list).
• Prompts for $(TargetFramework) for multi-targeted projects in interactive mode; errors with a --framework suggestion in non-interactive mode.
• Calls the ComputeAvailableDevices MSBuild target via RunCommandSelector and prints the available devices with a friendly dotnet test --device <id> example.
• Exits early without building, deploying, or running any tests.
• If the project has no ComputeAvailableDevices target, exits silently with success (matches dotnet run --list-devices).

Implementation notes

To keep the listing flow structurally identical to dotnet run --list-devices, HandleListDevices reuses a single RunCommandSelector instance for TrySelectTargetFramework -> InvalidateGlobalProperties -> TrySelectDevice(listDevices: true), the same sequence as RunCommand.TrySelectTargetFrameworkAndDeviceIfNeeded.

RunCommandSelector.TrySelectDevice gained a required commandName parameter so the listed/error example renders dotnet test --device <id> instead of the previously hard-coded dotnet run --device <id>. The two existing call sites (dotnet run, per-TFM device selection in tests) pass "dotnet run" and "dotnet test" respectively.

Code review surfaced a separate latent bug from #54295: when --device was used against a solution, the error told users to specify --framework even though that wouldn't fix the failure. Both --device and --list-devices now throw TestCommandUseProject (Specifying a project for 'dotnet test' should be via '--project'.).

Tests

Five new integration tests in GivenDotnetTestSelectsDevice:

• lists devices for a multi-TFM project with -f
• lists a single device when only one is available
• fails non-interactively when multi-targeted without -f
• exits silently for projects without ComputeAvailableDevices
• errors with the --project hint when run against a solution

The MTP help snapshot is also updated for the new --list-devices line.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

[Copilot]

Pull request overview

This PR extends the dotnet test Microsoft Testing Platform (MTP) command to support --list-devices, mirroring the existing dotnet run --list-devices behavior so MAUI/Android/iOS test projects can enumerate device IDs without building or running tests.

Changes:

• Added dotnet test --list-devices option and an early-exit listing flow that calls ComputeAvailableDevices via RunCommandSelector.
• Updated RunCommandSelector.TrySelectDevice to accept a commandName so the printed example uses the invoking command (dotnet run vs dotnet test).
• Added integration tests for listing behavior and updated the MTP help snapshot.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/dotnet.Tests/CommandTests/Test/snapshots/MTPHelpSnapshotTests.VerifyMTPHelpOutput.verified.txt	Updates dotnet test MTP help snapshot to include --list-devices.
test/dotnet.Tests/CommandTests/Test/GivenDotnetTestSelectsDevice.cs	Adds integration tests covering --list-devices scenarios (multi-TFM, single device, missing target, solution error).
src/Cli/Microsoft.DotNet.Cli.Definitions/Commands/Test/TestCommandDefinition.MicrosoftTestingPlatform.cs	Adds the --list-devices option to the MTP dotnet test command definition.
src/Cli/dotnet/Commands/Test/MTP/SolutionAndProjectUtility.cs	Passes commandName: "dotnet test" into device selection to render correct examples.
src/Cli/dotnet/Commands/Test/MTP/Options.cs	Extends BuildOptions to carry the ListDevices flag.
src/Cli/dotnet/Commands/Test/MTP/MSBuildUtility.cs	Wires --list-devices parse result into BuildOptions.
src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs	Implements the --list-devices early-exit flow and improves solution handling for --device.
src/Cli/dotnet/Commands/Run/RunCommandSelector.cs	Adds commandName parameter to TrySelectDevice and uses it in example output.
src/Cli/dotnet/Commands/Run/RunCommand.cs	Updates dotnet run to pass commandName: "dotnet run" into TrySelectDevice.

[Evangelink]

⚠️ Disclaimer: This review was produced by a code-review subagent of GitHub Copilot CLI acting as a testfx / Microsoft.Testing.Platform expert reviewer, posted under my identity. Treat findings as drafted suggestions to verify rather than authoritative statements from me personally. I will follow up on anything that lands.

Reviewed the diff against main. Focused on parity with dotnet run --list-devices, option wiring, exit codes, and the new HandleListDevices flow.

🐛 Bugs / Correctness

1. ListDevicesOption is missing Arity = ArgumentArity.Zero

File: src/Cli/Microsoft.DotNet.Cli.Definitions/Commands/Test/TestCommandDefinition.MicrosoftTestingPlatform.cs

public readonly Option<bool> ListDevicesOption = new("--list-devices")
{
    Description = CommandDefinitionStrings.CommandOptionListDevicesDescriptionForTest,
};

The dotnet run counterpart deliberately sets Arity = ArgumentArity.Zero (RunCommandDefinition.cs:57‑61), and so does every other boolean flag added to this very class — NoAnsiOption, ListTestsOption, NoLaunchProfileOption. Without Arity.Zero, an Option<bool> defaults to ZeroOrOne, meaning System.CommandLine will try to bind the next positional token (e.g. dotnet test --list-devices SomeProject.csproj) as the option value and fail bool parsing rather than treating it as a flag. The mirrored dotnet run option exists precisely to prevent that ambiguity — the test command should match.

⚠️ Behavioral / Parity Concerns

2. TrySelectTargetFramework still suggests "dotnet run --framework …" when invoked from dotnet test

File: src/Cli/dotnet/Commands/Run/RunCommandSelector.cs:280, reached from the new HandleListDevices.

The PR went out of its way to thread a commandName parameter through TrySelectDevice so the "Example: dotnet test --device …" hint is correct. The exact same hard‑coded "dotnet run --framework {frameworks[0]}" string survives in TrySelectTargetFramework and now fires from two dotnet test code paths:

• HandleListDevices → selector.TrySelectTargetFramework(...) (new in this PR)
• HandleDeviceWithTargetFrameworkSelection → RunCommandSelector.TrySelectTargetFramework(...) (pre‑existing, untouched here)

So a user running dotnet test --list-devices (or dotnet test --device) against a multi‑TFM project in a non‑interactive shell still sees:

Example: dotnet run --framework net10.0-android

This is exactly the latent bug the PR claims to be cleaning up for --device on a solution. The new test ItFailsToListDevicesWhenMultipleTargetFrameworks_InNonInteractiveMode only asserts the first error string and therefore doesn't catch it. Consider plumbing commandName through TrySelectTargetFramework (and/or its static overload) the same way the PR did for TrySelectDevice, and adding a stderr.Should().NotContain("dotnet run") assertion.

3. --no-build silently implies --no-restore for device discovery in dotnet test, but not in dotnet run

File: MicrosoftTestingPlatformTestCommand.cs

noRestore: buildOptions.HasNoRestore || buildOptions.HasNoBuild,

RunCommand.TrySelectTargetFrameworkAndDeviceIfNeeded passes only NoRestore (RunCommand.cs:303). The MTP version conflates --no-build with --no-restore for the ComputeAvailableDevices evaluation. This matches the pre‑existing SelectDeviceForTfm in SolutionAndProjectUtility.cs:429, so it's at least internally consistent, but it diverges from the stated "mirror dotnet run --list-devices" goal and means that on a clean project dotnet test --list-devices --no-build will skip the restore that dotnet run --list-devices --no-build would perform — and ComputeAvailableDevices may then fail on a missing obj/project.assets.json. Worth a deliberate decision rather than an accidental one.

4. --list-devices silently swallows --device and --test-modules

File: MicrosoftTestingPlatformTestCommand.cs

The PR (correctly) added a hard error for --list-devices + --list-tests. But:

• --list-devices --device X → ListDevices short‑circuits at the top of Run, Device is silently ignored. dotnet run has the same behavior, so this is parity, but it's worth being explicit (either a third mutually‑exclusive check or a documented "device is ignored").
• --list-devices --test-modules foo → ValidationUtility.ValidateOptionsIrrelevantToModulesFilter only screens Configuration/Framework/Architecture/OS/Runtime, not Device/ListDevices. HandleListDevices then calls ValidateBuildPathOptions, which ignores TestModules entirely and falls back to scanning the current directory for a project — surprising and almost certainly not what the user asked for. Pre‑existing for --device too, but the PR is a natural place to tighten it.

🧪 Test Gaps

The five new tests cover the happy path and the "must‑error" cases reasonably well, but a few high‑value cases are missing:

• No regression test for the "dotnet test --device" hint added in the new diff. The positive ItListsDevicesAndExits asserts StdOut.Should().Contain("dotnet test --device") (good), but there's no equivalent for the non‑interactive multi‑device fallback path. Worth a test that exercises the else branch of TrySelectDevice so the rendered example doesn't silently regress.
• --list-devices on .slnf: only .slnx is tested (ItErrorsWhenListingDevicesForSolution). .slnf is in CliConstants.SolutionExtensions so it should take the same branch, but the --solution parsing for .slnf has had its own bugs historically.
• --list-devices with explicit -f on a multi‑TFM device project: confirms the "non‑interactive TFM prompt failure" doesn't fire when the user is doing the right thing. The current ItListsDevicesAndExits uses ToolsetInfo.CurrentTargetFramework which the asset declares as the only TFM, so we never actually validate the "two TFMs + explicit --framework" path that MAUI users will hit in practice.
• --list-devices + --test-modules (see concern 4).
• --list-devices + --device (see concern 4) — assert that --device is ignored / no error, so any future stricter behavior is a deliberate decision.

💡 Optional Observations

• HandleListDevices reproduces the isInteractive heuristic !Console.IsOutputRedirected && !new CIEnvironmentDetectorForTelemetry().IsCIEnvironment(). dotnet run instead honors the explicit --interactive option (RunCommand.cs:948). The MTP command does expose interactivity differently, so this matches the surrounding code, but it does mean dotnet test --interactive --list-devices on a TTY behaves identically to dotnet test --list-devices. Possibly fine, possibly a future paper cut.
• commandName is a required positional string placed at the end of TrySelectDevice after three out parameters. All call sites use named‑arg syntax so it works, but string commandName = "dotnet" with explicit overrides at the two callers would be a smaller, less surprising signature change — and would defuse any future caller (e.g., a unit test using positional args) accidentally compiling against a stale shape. Not worth blocking on.
• The --list-devices/--list-tests mutual‑exclusion message lives in CliCommandStrings.resx (runtime layer). Correct choice — the check executes in MicrosoftTestingPlatformTestCommand.Run, not at definition time. ✅
• Placeholders in the two new CommandOptionListDevicesDescriptionForTest / CommandOptionDeviceDescriptionForTest strings: both are plain sentences with no {0}/{1}, so no format‑string mismatch risk. ✅

[jonathanpeppers]

Thanks for the thorough review! Pushed a follow-up commit (squashed into the single commit on the branch). Summary of what I addressed:

Adopted:

• Templates: Deterministic builds should be on by default #1 Arity = ArgumentArity.Zero on ListDevicesOption - added. Matches every other bool option in TestCommandDefinition.MicrosoftTestingPlatform.cs and prevents the option from greedily consuming the next positional token.
• AssemblyInfo: Determine which properties will be generated and their msbuild mappings #2 Hard-coded "dotnet run --framework" in error output - fixed. RunCommandSelector now takes commandName via its constructor (stored as _commandName), and the static TrySelectTargetFramework overload takes it as a parameter. Callers from MicrosoftTestingPlatformTestCommand / SolutionAndProjectUtility pass "dotnet test"; the RunCommand callers pass "dotnet run". Strengthened the existing ItFailsToListDevicesWhenMultipleTargetFrameworks_InNonInteractiveMode test to assert stderr contains "dotnet test --framework" and does not contain "dotnet run --framework".
• Added netci for jenkins #4 --list-devices / --device swallowing --test-modules - added explicit validation in MicrosoftTestingPlatformTestCommand.Run that fails fast with a clear error (new resx string CmdDeviceOptionsRequireProject) when these are combined. New test ItErrorsWhenListDevicesIsCombinedWithTestModules covers it.

Skipped (with rationale):

• A Basic Solution Structure to Enable Infrastructure Development #3 noRestore conflated with HasNoBuild - this matches the existing pattern in SelectDeviceForTfm and dotnet run's own behavior. Decoupling them would be a behavior change worth its own PR rather than rolling into this one.
• Test gap on .slnf solution filters - .slnx coverage exercises the same SolutionAndProjectUtility path; .slnf adds marginal value here.
• Test gap on --list-devices --device combination - parity with dotnet run, which has the same shape.
• Optional Observations - all noted by the reviewer as non-blocking; left as-is to keep the diff focused.