`dotnet test --arch x64` fails

[richlander]

Looks like crossing of the streams, using the Arm64 SDK to test with x64.

Setup:

dotnet new xunit -o tests

rich@MacBook-Air-M1-2020 tests % /usr/local/share/dotnet/x64/dotnet test
  Determining projects to restore...
  Restored /Users/rich/tests/tests.csproj (in 903 ms).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tests -> /Users/rich/tests/bin/Debug/net6.0/tests.dll
Test run for /Users/rich/tests/bin/Debug/net6.0/tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - /Users/rich/tests/bin/Debug/net6.0/tests.dll (net6.0)
rich@MacBook-Air-M1-2020 tests % dotnet test --arch x64                 
  Determining projects to restore...
  Restored /Users/rich/tests/tests.csproj (in 959 ms).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tests -> /Users/rich/tests/bin/Debug/net6.0/osx-x64/tests.dll
Test run for /Users/rich/tests/bin/Debug/net6.0/osx-x64/tests.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.29] Exception discovering tests from tests: System.BadImageFormatException: Could not load file or assembly 'tests, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'. An attempt was made to load a program with an incorrect format.

File name: 'tests, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(ObjectHandleOnStack assemblyName, ObjectHandleOnStack requestingAssembly, StackCrawlMarkHandle stackMark, Boolean throwOnFileNotFound, ObjectHandleOnStack assemblyLoadContext, ObjectHandleOnStack retAssembly)
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, RuntimeAssembly requestingAssembly, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Xunit.Sdk.ReflectionAssemblyInfo..ctor(String assemblyFileName) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Reflection\ReflectionAssemblyInfo.cs:line 31
--- End of stack trace from previous location ---
   at ExceptionExtensions.RethrowWithNoStackTraceLoss(Exception ex) in C:\Dev\xunit\xunit\src\common\ExceptionExtensions.cs:line 27
   at Xunit.AppDomainManager_NoAppDomain.CreateObject[TObject](AssemblyName assemblyName, String typeName, Object[] args) in C:\Dev\xunit\xunit\src\xunit.runner.utility\AppDomain\AppDomainManager_NoAppDomain.cs:line 30
   at Xunit.Xunit2Discoverer..ctor(AppDomainSupport appDomainSupport, ISourceInformationProvider sourceInformationProvider, IAssemblyInfo assemblyInfo, String assemblyFileName, String xunitExecutionAssemblyPath, String configFileName, Boolean shadowCopy, String shadowCopyFolder, IMessageSink diagnosticMessageSink, Boolean verifyAssembliesOnDisk) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\v2\Xunit2Discoverer.cs:line 102
   at Xunit.Xunit2Discoverer..ctor(AppDomainSupport appDomainSupport, ISourceInformationProvider sourceInformationProvider, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, IMessageSink diagnosticMessageSink, Boolean verifyAssembliesOnDisk) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\v2\Xunit2Discoverer.cs:line 62
   at Xunit.Xunit2..ctor(AppDomainSupport appDomainSupport, ISourceInformationProvider sourceInformationProvider, String assemblyFileName, String configFileName, Boolean shadowCopy, String shadowCopyFolder, IMessageSink diagnosticMessageSink, Boolean verifyTestAssemblyExists) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\v2\Xunit2.cs:line 41
   at Xunit.XunitFrontController.CreateInnerController() in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:line 144
   at Xunit.XunitFrontController.EnsureInitialized() in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:line 163
   at Xunit.XunitFrontController.get_InnerController() in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:line 103
   at Xunit.XunitFrontController.get_TargetFramework() in C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:line 111
   at Xunit.Runner.VisualStudio.VsTestRunner.DiscoverTestsInSource[TVisitor](XunitFrontController framework, LoggerHelper logger, TestPlatformContext testPlatformContext, RunSettings runSettings, Func`4 visitorFactory, Action`4 visitComplete, String assemblyFileName, Boolean shadowCopy, TestAssemblyConfiguration configuration) in /_/src/xunit.runner.visualstudio/VsTestRunner.cs:line 283
No test is available in /Users/rich/tests/bin/Debug/net6.0/osx-x64/tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Same thing in reverse.

rich@MacBook-Air-M1-2020 tests % dotnet --info | grep RID
 RID:         osx-x64
rich@MacBook-Air-M1-2020 tests % dotnet test --arch arm64
Resolving the current runtime identifier failed.

Similar thing with .NET Core 3.1.

rich@MacBook-Air-M1-2020 tests % cat tests.csproj | grep Target
    <TargetFramework>netcoreapp3.1</TargetFramework>
rich@MacBook-Air-M1-2020 tests % dotnet --info | grep RID
 RID:         osx-arm64
rich@MacBook-Air-M1-2020 tests % dotnet test
  Determining projects to restore...
  All projects are up-to-date for restore.
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tests -> /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll
Test run for /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll (.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Testhost process exited with error: It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '3.1.0' (arm64) was not found.
  - The following frameworks were found:
      6.0.0-rc.2.21470.23 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
You can resolve the problem by installing the specified framework and/or SDK.
The specified framework can be found at:
  - https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=3.1.0&arch=arm64&rid=osx.11.1-arm64
. Please check the diagnostic logs for more information.

Test Run Aborted.
rich@MacBook-Air-M1-2020 tests % dotnet test -a x64
  Determining projects to restore...
  All projects are up-to-date for restore.
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tests -> /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll
Test run for /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll (.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The path '/Users/rich/tests/x64' specified in the 'TestAdapterPath' is invalid. Error: The custom test adapter search path provided was not found, provide a valid path and try again.
The path '/Users/rich/tests/x64' specified in the 'TestAdapterPath' is invalid. Error: The custom test adapter search path provided was not found, provide a valid path and try again.
rich@MacBook-Air-M1-2020 tests % dotnet test --arch x64
  Determining projects to restore...
  Restored /Users/rich/tests/tests.csproj (in 1.13 sec).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tests -> /Users/rich/tests/bin/Debug/netcoreapp3.1/osx-x64/tests.dll
Test run for /Users/rich/tests/bin/Debug/netcoreapp3.1/osx-x64/tests.dll (.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Testhost process exited with error: It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '3.1.0' (arm64) was not found.
  - The following frameworks were found:
      6.0.0-rc.2.21470.23 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
You can resolve the problem by installing the specified framework and/or SDK.
The specified framework can be found at:
  - https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=3.1.0&arch=arm64&rid=osx.11.1-arm64
. Please check the diagnostic logs for more information.

Test Run Aborted.
rich@MacBook-Air-M1-2020 tests % /usr/local/share/dotnet/x64/dotnet test
  Determining projects to restore...
  Restored /Users/rich/tests/tests.csproj (in 705 ms).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  tests -> /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll
Test run for /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll (.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - /Users/rich/tests/bin/Debug/netcoreapp3.1/tests.dll (netcoreapp3.1)

[sfoslund]

@nohwnd for the dotnet test team

[vitek-karas]

This is more or less expected right now.

We solve this problem on Windows for x86/x64 by actually building the apphost testhost for the right architecture (just like dotnet build does) and then running the test using that executable.

Anywhere else (so osx-* or win-arm64) we fall back to running the tests via the muxer, so dotnet.exe testhost.dll - using the dotnet.exe which was used to start the SDK/CLI command.
So in the first example above where osx-arm64 SDK is trying to run osx-x64 tests, the test process is started with osx-arm64 dotnet testhost.dll but pointing to a build of the tests for osx-x64. This fails, probably because the test.dll has in the PE architecture x64, which will fail the architecture check on arm64 runtime. In any case, if it worked it would still be broken since it would run the test on arm64 runtime.

The advantage of using testhost executable is that it's really just an apphost, which will find the right runtime for itself. So it would effectively work due to same functionality we made dotnet run work.

I don't know what it would take to use the testhost executable on all platforms/archs, but it should fix the problem.

[MarcoRossignoli]

Fixed in 6.0.200/7.0.100