Use runtest.py to run tests for all platforms (#19213)

Change build-test.sh to always build the xunit wrappers. Before it would drop a token and check the existence of the token.

Unify x64 linux/OSX/Windows excludes into one file, issues.targets. Includes different locations in the file which show where to put excludes.

Remove all target specific aspects of issues.targets, all tests are excluded now via wildcard, this allows expanding to .cmd and .sh based on the built platform.

Unify path separators to forward slash(/) in issues.targets to support both platforms

Clean up issues.targets by removing long standing exclude tests, specifically tests that have been excluded due to missing features like rva_statics.
Add DisableProjectBuild to tests which have been removed from issues.targets

Conditionally add DisableProjectBuild to tests which have been marked as unsupported on unix. This is mostly a port of the unsupportedOnUnix.txt list. Instead of excluding the tests, unix will simply not build them. If tests are built on windows, they will be run but they will return pass, the test wrapper will check return instantly.

All exclusions ported to issues.targets for linux targets.
Expand runtest.py, this includes simple issues that made it past the original CR. In addition it adds more optional features to help with inner loop dev work such as: creating a repro folder under bin/repro/.. which sets up the env and calls the failing test. In addition a launch.json will now be created under bin/repro/.. which can be used to easily debug using vscode. More logging, such as printing failures, longest running tests ect.
Initial excludes ported for arm64 windows

Arm64 linux, armhf unix excludes and enables running runtest.sh for these targets.
arm64 windows and arm32 windows excludes and enables running runtest.cmd on arm64 targets
init-tools.sh changes to pull armhf and aarch64 dotnetcli

init-tools.cmd changes to pull x86 packages for dotnetcli for arm64 windows
runtest.cmd for almost all scenarios will call runtest.py
runtest.sh for almsot all scenarios will call runtest.py
Removes all logic for running tests using runtest.sh

Documentation/building/test-configuration.md

@@ -39,15 +39,17 @@ Test cases are categorized by priority level. The most important subset should b
     ```
 * Disable building of a test by conditionally setting the `<DisableProjectBuild>` property.
 	* e.g. `<DisableProjectBuild Condition=" '$(Platform)' == 'arm64' ">true</DisableProjectBuild>`
+* Exclude test from GCStress runs by adding the following to the csproj:
+    * `<GCStressIncompatible>true</GCStressIncompatible>`
+* Exclude test from JIT stress runs runs by adding the following to the csproj:
+    * `<JitOptimizationSensitive>true</JitOptimizationSensitive>`
 * Add NuGet/MyGet references by updating the following [test project](https://github.com/dotnet/coreclr/blob/master/tests/src/Common/test_dependencies/test_dependencies.csproj).
 * Build against the `mscorlib` facade by adding `<ReferenceLocalMscorlib>true</ReferenceLocalMscorlib>` to the test project.
 * Update relevent exclusion lists:
-  * Note that there are two build pipelines Jenkin's CI and nightly Helix - both must be updated for expected exclusion.
-    * Jenkin's CI build - see the associated `*.txt` files under `tests/` (e.g. `tests/testsUnsupportedOutsideWindows.txt`).
-    * Official nightly Helix build - see the `tests/issues.targets` file.
-  * ARM/ARM64
-    * `tests/arm/Tests.lst` and `tests/arm64/Tests.lst` are used to define the tests to run due to limitations with XUnit.
-    * These files can be manually edited or the generated using `tests/scripts/lst_creator.py`.
+  There are currently three different exclude lists. Tests.lst is currently only used by CI.
+    - `tests/issues.targets`
+    - `tests/arm/Tests.lst` - Used by Windows arm32 testing
+    - `tests/arm64/Tests.lst` - Used by Windows arm64 testing
 
 ### Creating a C# test project
 

Documentation/building/unix-test-instructions.md

@@ -81,50 +81,15 @@ sudo apt-get install libunwind8:armhf libunwind8-dev:armhf libicu-dev:armhf libl
 The following instructions assume that on the Unix machine:
 - The CoreCLR repo is cloned at `/mnt/coreclr`
 
-If DotNet is unsupported
-- The CoreFX repo is cloned at `/mnt/corefx`
-- The other platform's clone of the CoreCLR repo is mounted at `/media/coreclr`
-
-The following steps are different if DotNet is supported or not on your arch and os.
-
-### DotNet is supported
-
-build-test.sh will have setup the Core_Root directory correctly after the test build. If this was either skipped or needs to be regenerated use:
-
->`build-test.sh generatelayoutonly`
-
-To run the tests run with the --coreOverlayDir path
+build-test.sh will have setup the Core_Root directory correctly after the test build.
 
 ```bash
-~/coreclr$ tests/runtest.sh
-    --testRootDir=/mnt/coreclr/bin/tests/Linux.x64.Debug
-    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
-    --coreOverlayDir=/mnt/coreclr/bin/tests/Linux.x64.Debug/Tests/Core_Root
-    --copyNativeTestBin
+~/coreclr$ tests/runtest.sh x64 checked
 ```
 
-### DotNet is not supported
-
-Tests need to be built on another platform and copied over to the Unix machine for testing. Copy the test build over to the Unix machine:
-
-> `cp --recursive /media/coreclr/bin/tests/Windows_NT.x64.Debug /mnt/test/`
-
-See `runtest.sh` usage information:
-
-> `/mnt/coreclr$ tests/runtest.sh --help`
-
-Run tests (`Debug` may be replaced with `Release` or `Checked`, depending on which Configuration you've built):
+Please use the following command for help.
 
-```bash
-/mnt/coreclr$ tests/runtest.sh
-    --testRootDir=/mnt/test/Windows_NT.x64.Debug
-    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
-    --coreClrBinDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
-    --mscorlibDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
-    --coreFxBinDir=/mnt/corefx/bin/runtime/netcoreapp-Linux-Debug-x64
-```
-
-The method above will copy dependencies from the set of directories provided to create an 'overlay' directory.
+>./tests/runtest.sh -h
 
 ### Results
 
@@ -134,13 +99,21 @@ Test results will go into:
 
 ### Unsupported and temporarily disabled tests
 
-These tests are skipped by default:
-- Tests that are not supported outside Windows, are listed in:
-    > `~/coreclr/tests/testsUnsupportedOutsideWindows.txt`
-- Tests that are temporarily disabled outside Windows due to unexpected failures (pending investigation), are listed in:
-    > `~/coreclr/tests/testsFailingOutsideWindows.txt`
+Unsupported tests outside of Windows have two annotations in their csproj to
+ignore them when run.
+
+```
+<TestUnsupportedOutsideWindows>true</TestUnsupportedOutsideWindows>
+```
+
+This will write in the bash target to skip the test by returning a passing value if run outside Windows.
+
+In addition:
+```
+<DisableProjectBuild Condition="'$(TargetsUnix)' == 'true'">true</DisableProjectBuild>
+```
 
-To run only the set of temporarily disabled tests, pass in the `--runFailingTestsOnly` argument to `runtest.sh`.
+Is used to disable the build, that way if building on unix cycles are saved building/running.
 
 PAL tests
 ---------

Documentation/building/windows-test-instructions.md

@@ -21,9 +21,9 @@ This will use `crossgen.exe` to precompile the test executables before they are
 
 ## Building Other Priority Tests
 
-    C:\git\coreclr>build-test.cmd -priority=2
+    C:\git\coreclr>build-test.cmd -priority=1
 
-The number '2' is just an example. The default value (if no priority is specified) is 0. To clarify, if '2' is specified, all tests with CLRTestPriorty 0, 1 **and** 2 will be built and consequently run.
+The number '1' is just an example. The default value (if no priority is specified) is 0. To clarify, if '1' is specified, all tests with CLRTestPriorty 0 **and** 1 will be built and consequently run.
 
 ## Examples
 
@@ -35,7 +35,7 @@ To run a clean, priority 1, crossgen test pass:
 
 ### Building Individual Tests
 
-Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast once
+Note: build-test.cmd or build.cmd skipnative needs to be run atleast once
 
 * Native Test: Build the generated Visual Studio solution or make file corresponding to Test cmake file.
 
@@ -47,7 +47,7 @@ Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast
 
 For example to run all of the tests using your checked build:
 
-     <repo_root>\tests\runtest.cmd -checked
+     <repo_root>\tests\runtest.cmd checked
 
 This will generate a report named as `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT__x64__Checked.html`) in the subdirectory `<repo_root>\bin\Logs`. Any tests that failed will be listed in `TestRunResults_Windows_NT__x64__Checked.err`.
 

build-test.sh

@@ -289,22 +289,18 @@ build_Tests()
     if [ $__BuildTestWrappers -ne -0 ]; then
         echo "${__MsgPrefix}Creating test wrappers..."
 
-        __XUnitWrapperBuiltMarker=${__TestBinDir}/xunit_wrapper_build
+        # Always create the test wrappers and set the exclude file.
+        export __Exclude="$__TestDir/issues.targets"
+        echo "Exclude set to $__TestDir/issues.targets"
+        build_Tests_internal "Tests_XunitWrapper" "$__ProjectDir/tests/runtest.proj" "Test Xunit Wrapper" "-BuildWrappers" "-MsBuildEventLogging= " "-TargetsWindows=false"
 
-        if [ ! -f $__XUnitWrapperBuiltMarker ]; then
-
-            build_MSBuild_projects "Tests_XunitWrapper" "$__ProjectDir/tests/runtest.proj" "Test Xunit Wrapper" "-BuildWrappers" "-MsBuildEventLogging= " "-TargetsWindows=false"
-
-            if [ $? -ne 0 ]; then
-                echo "${__MsgPrefix}Error: build failed. Refer to the build log files for details (above)"
-                exit 1
-            else
-                echo "XUnit Wrappers have been built."
-                echo "Create marker \"${__XUnitWrapperBuiltMarker}\""
-                touch $__XUnitWrapperBuiltMarker
-            fi
+        if [ $? -ne 0 ]; then
+            echo "${__MsgPrefix}Error: build failed. Refer to the build log files for details (above)"
+            exit 1
         else
-            echo "XUnit Wrappers had been built before."
+            echo "XUnit Wrappers have been built."
+            echo "Create marker \"${__XUnitWrapperBuiltMarker}\""
+            touch $__XUnitWrapperBuiltMarker
         fi
     fi
 

init-tools.cmd

@@ -47,9 +47,33 @@ echo Running %0 > "%INIT_TOOLS_LOG%"
 set /p DOTNET_VERSION=< "%~dp0DotnetCLIVersion.txt"
 if exist "%DOTNET_CMD%" goto :afterdotnetrestore
 
+REM Use x86 tools on arm64 and x86.
+REM arm32 host is not currently supported, please crossbuild.
+if /i "%PROCESSOR_ARCHITECTURE%" == "arm" (
+  echo "Error, arm32 arch not supported for build tools."
+  exit /b 1
+)
+
+if /i "%PROCESSOR_ARCHITECTURE%" == "amd64" (
+  set _Arch="x64"
+  goto ArchSet
+)
+
+REM If this is not amd64 and not arm, then we should be running on arm64 or x86
+REM either way we can (and should) use the x86 dotnet cli
+REM
+REM TODO: consume native arm64 toolset, blocked by official arm64 windows cli
+REM     : release. See https://github.com/dotnet/coreclr/issues/19614 for more
+REM     : information
+set _Arch="x86"
+
+echo "init-tools.cmd: Setting arch to %_Arch% for build tools"
+
+:ArchSet
+
 echo Installing dotnet cli...
 if NOT exist "%DOTNET_PATH%" mkdir "%DOTNET_PATH%"
-set DOTNET_ZIP_NAME=dotnet-sdk-%DOTNET_VERSION%-win-x64.zip
+set DOTNET_ZIP_NAME=dotnet-sdk-%DOTNET_VERSION%-win-%_Arch%.zip
 set DOTNET_REMOTE_PATH=https://dotnetcli.azureedge.net/dotnet/Sdk/%DOTNET_VERSION%/%DOTNET_ZIP_NAME%
 set DOTNET_LOCAL_PATH=%DOTNET_PATH%%DOTNET_ZIP_NAME%
 echo Installing '%DOTNET_REMOTE_PATH%' to '%DOTNET_LOCAL_PATH%' >> "%INIT_TOOLS_LOG%"

init-tools.sh

@@ -71,7 +71,17 @@ if [ ! -e "$__DOTNET_PATH" ]; then
             echo "Warning: build not supported on 32 bit Unix"
         fi
 
-        __PKG_ARCH=x64
+        if [ "$(uname -m)" = "armhf" ] || [ "$(uname -m)" = "arm64" ] || [ "$(uname -m)" = "aarch64" ];  then
+            if [ "$(uname -m)" = "armhf" ]; then
+                __PKG_ARCH=arm
+            fi
+
+            if [ "$(uname -m)" = "arm64" ] || [ "$(uname -m)" = "aarch64" ]; then
+                __PKG_ARCH=arm64
+            fi
+        else
+            __PKG_ARCH=x64
+        fi
 
         OSName=$(uname -s)
         case $OSName in
@@ -171,26 +181,30 @@ if [ -n "${DotNetBootstrapCliTarPath-}" ]; then
     unset ILASMCOMPILER_VERSION
 fi
 
-echo "Initializing BuildTools..."
-echo "Running: $__BUILD_TOOLS_PATH/init-tools.sh $__scriptpath $__DOTNET_CMD $__TOOLRUNTIME_DIR $__PACKAGES_DIR" >> "$__init_tools_log"
-
-# Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
-chmod +x "$__BUILD_TOOLS_PATH/init-tools.sh"
-"$__BUILD_TOOLS_PATH/init-tools.sh" "$__scriptpath" "$__DOTNET_CMD" "$__TOOLRUNTIME_DIR" "$__PACKAGES_DIR" >> "$__init_tools_log"
-if [ "$?" != "0" ]; then
-    echo "ERROR: An error occurred when trying to initialize the tools." 1>&2
-    display_error_message
-    exit 1
-fi
-
-echo "Making all .sh files executable under Tools."
-# Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
-ls "$__scriptpath/Tools/"*.sh | xargs chmod +x
-ls "$__scriptpath/Tools/scripts/docker/"*.sh | xargs chmod +x
+# Build tools only supported on x64
+if [ "${__PKG_ARCH}" != "x64" ] &&  [ "${__PKG_ARCH}" != "arm" ]; then
+    echo "Skipped installing build tools."
+else
+    echo "Initializing BuildTools..."
+    echo "Running: $__BUILD_TOOLS_PATH/init-tools.sh $__scriptpath $__DOTNET_CMD $__TOOLRUNTIME_DIR $__PACKAGES_DIR" >> "$__init_tools_log"
+
+    # Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
+    chmod +x "$__BUILD_TOOLS_PATH/init-tools.sh"
+    "$__BUILD_TOOLS_PATH/init-tools.sh" "$__scriptpath" "$__DOTNET_CMD" "$__TOOLRUNTIME_DIR" "$__PACKAGES_DIR" >> "$__init_tools_log"
+    if [ "$?" != "0" ]; then
+        echo "ERROR: An error occurred when trying to initialize the tools." 1>&2
+        display_error_message
+        exit 1
+    fi
 
-"$__scriptpath/Tools/crossgen.sh" "$__scriptpath/Tools" $__PKG_RID
+    echo "Making all .sh files executable under Tools."
+    # Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
+    ls "$__scriptpath/Tools/"*.sh | xargs chmod +x
+    ls "$__scriptpath/Tools/scripts/docker/"*.sh | xargs chmod +x
 
-mkdir -p "$(dirname "$__BUILD_TOOLS_SEMAPHORE")" && touch "$__BUILD_TOOLS_SEMAPHORE"
+    "$__scriptpath/Tools/crossgen.sh" "$__scriptpath/Tools" $__PKG_RID
 
-echo "Done initializing tools."
+    mkdir -p "$(dirname "$__BUILD_TOOLS_SEMAPHORE")" && touch "$__BUILD_TOOLS_SEMAPHORE"
 
+    echo "Done initializing tools."
+fi

tests/arm/Tests.lst

@@ -53204,14 +53204,6 @@ MaxAllowedDurationSeconds=600
 Categories=EXPECTED_PASS;Pri1
 HostStyle=0
 
-[b12011.cmd_6691]
-RelativePath=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011\b12011.cmd
-WorkingDir=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011
-Expected=0
-MaxAllowedDurationSeconds=600
-Categories=EXPECTED_PASS
-HostStyle=0
-
 [FlowControlNext.cmd_6692]
 RelativePath=CoreMangLib\cti\system\reflection\emit\flowcontrol\FlowControlNext\FlowControlNext.cmd
 WorkingDir=CoreMangLib\cti\system\reflection\emit\flowcontrol\FlowControlNext

tests/arm64/Tests.lst

@@ -70244,14 +70244,6 @@ MaxAllowedDurationSeconds=600
 Categories=JIT;EXPECTED_PASS
 HostStyle=0
 
-[b12011.cmd_9160]
-RelativePath=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011\b12011.cmd
-WorkingDir=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011
-Expected=0
-MaxAllowedDurationSeconds=600
-Categories=EXPECTED_PASS
-HostStyle=0
-
 [b14355.cmd_9161]
 RelativePath=JIT\Regression\VS-ia64-JIT\V1.2-M02\b14355\b14355\b14355.cmd
 WorkingDir=JIT\Regression\VS-ia64-JIT\V1.2-M02\b14355\b14355