Make Ninja the default CMake generator on Windows for the repo (#49715)

Co-authored-by: Juan Hoyos <juan.hoyos@microsoft.com>

docs/workflow/debugging/coreclr/debugging.md

@@ -8,28 +8,32 @@ SOS has moved to the diagnostics repo. For more information on SOS, installation
 Debugging CoreCLR on Windows
 ============================
 
-1. Perform a build of the repo.
-2. Open solution \<reporoot\>\artifacts\obj\coreclr\windows.\<platform\>.\<configuration\>\CoreCLR.sln in Visual Studio. \<platform\> and \<configuration\> are based
+1. Open the CoreCLR solution in Visual Studio.
+   - Method 1: Use the build scripts to open the solution.
+      1. Run `./build.cmd -vs coreclr.sln -a <platform> -c <configuration>`. This will create and launch the CoreCLR solution in VS for the specified architecture and configuration.
+   - Method 2: Manually build and open the solution.
+      1. Perform a build of the repo with the `-msbuild` flag.
+      2. Open solution `\<reporoot>\artifacts\obj\coreclr\windows.\<platform>.\<configuration>\ide\CoreCLR.sln` in Visual Studio. `<platform>` and `<configuration>` are based
     on type of build you did. By default they are 'x64' and 'Debug'.
-3. Right-click the INSTALL project and choose ‘Set as StartUp Project’
-4. Bring up the properties page for the INSTALL project
-5. Select Configuration Properties->Debugging from the left side tree control
-6. Set Command=`$(SolutionDir)\..\..\..\bin\coreclr\windows.$(Platform).$(Configuration)\corerun.exe`
+2. Right-click the INSTALL project and choose ‘Set as StartUp Project’
+3. Bring up the properties page for the INSTALL project
+4. Select Configuration Properties->Debugging from the left side tree control
+5. Set Command=`$(SolutionDir)\..\..\..\bin\coreclr\windows.$(Platform).$(Configuration)\corerun.exe`
     1. This points to the folder where the built runtime binaries are present.
-7. Set Command Arguments=`<managed app you wish to run>` (e.g. HelloWorld.dll)
-8. Set Working Directory=`$(SolutionDir)\..\..\..\bin\coreclr\windows.$(Platform).$(Configuration)`
+6. Set Command Arguments=`<managed app you wish to run>` (e.g. HelloWorld.dll)
+7. Set Working Directory=`$(SolutionDir)\..\..\..\bin\coreclr\windows.$(Platform).$(Configuration)`
     1. This points to the folder containing CoreCLR binaries.
-9. Set Environment=`CORE_LIBRARIES=$(SolutionDir)\..\..\..\bin\runtime\<current tfm>-windows-$(Configuration)-$(Platform)`,
+8. Set Environment=`CORE_LIBRARIES=$(SolutionDir)\..\..\..\bin\runtime\<current tfm>-windows-$(Configuration)-$(Platform)`,
     where '\<current tfm\>' is the target framework of current branch, for example `netcoreapp3.1` `net5.0`.
     1. This points to the folder containing core libraries except `System.Private.CoreLib`.
     2. This step can be skipped if you are debugging CLR tests that references only `System.Private.CoreLib`.
     Otherwise, it's required to debug a realworld application that references anything else, including `System.Runtime`.
-10. Right-click the INSTALL project and choose 'Build'
+9.  Right-click the INSTALL project and choose 'Build'
     1. This will load necessary information from cmake to Visual Studio.
-11. Press F11 to start debugging at wmain in corerun (or set a breakpoint in source and press F5 to run to it)
+10. Press F11 to start debugging at wmain in corerun (or set a breakpoint in source and press F5 to run to it)
     1. As an example, set a breakpoint for the EEStartup function in ceemain.cpp to break into CoreCLR startup.
 
-Steps 1-10 only need to be done once, and then (11) can be repeated whenever you want to start debugging. The above can be done with Visual Studio 2019 as writing.
+Steps 1-9 only need to be done once as long as there's been no changes to the CMake files in the repository. Afterwards, step 10 can be repeated whenever you want to start debugging. The above can be done with Visual Studio 2019 as writing.
 Keeping with latest version of Visual Studio is recommended.
 
 ### Using SOS with windbg or cdb on Windows ###

docs/workflow/editing-and-debugging.md

@@ -19,19 +19,20 @@ The repository has a number of Visual Studio Solutions files (`*.sln`) that are
 
    * `src\coreclr\System.Private.CoreLib\System.Private.CorLib.sln` - This solution is for all managed (C#) code that is defined
    in the runtime itself.   This is all class library support of one form or another.
-   * `artifacts\obj\coreclr\windows.<Arch>.<BuildType>\CoreCLR.sln` - this solution contains most native (C++) projects
+   * `artifacts\obj\coreclr\windows.<Arch>.<BuildType>\ide\CoreCLR.sln` - this solution contains most native (C++) projects
    associated with the repository, including
      * `coreclr` - This is the main runtime DLL (the GC, class loader, interop are all here)
      * `corjit` - This is the Just In Time (JIT) compiler that compiles .NET Intermediate language to native code.
      * `corerun` - This is the simple host program that can run a .NET application
      * `crossgen` - This is the host program that runs the JIT compiler and produces .NET Native images (`*.ni.dll`)
      for C# code.
+     * This project can be automatically generated and opened in Visual Studio by running `./build.cmd -vs CoreCLR.sln -a <Arch> -c <BuildType>` from the root of the repository.
 
 Thus opening one of these two solution files (double clicking on them in Explorer) is typically all you need
 to do most editing.
 
 Notice that the CoreCLR solution is under the `artifacts` directory.  This is because it is created as part of the build.
-Thus you can only launch this solution after you have built at least once.
+Thus you can only launch this solution after you have built at least once with the `-msbuild` flag or run the `./build.cmd -vs CoreCLR.sln` command line with the specified architecture and configuration.
 
 * See [Debugging CoreCLR](debugging/coreclr/debugging.md)
 

eng/build.ps1

@@ -17,6 +17,7 @@ Param(
   [ValidateSet("Debug","Release")][string][Alias('lc')]$librariesConfiguration,
   [ValidateSet("CoreCLR","Mono")][string][Alias('rf')]$runtimeFlavor,
   [switch]$ninja,
+  [switch]$msbuild,
   [string]$cmakeargs,
   [switch]$pgoinstrument,
   [Parameter(ValueFromRemainingArguments=$true)][String[]]$properties
@@ -51,7 +52,7 @@ function Get-Help() {
   Write-Host "                                 [Default: Minimal]"
   Write-Host "  -vs                            Open the solution with Visual Studio using the locally acquired SDK."
   Write-Host "                                 Path or any project or solution name is accepted."
-  Write-Host "                                 (Example: -vs Microsoft.CSharp)"
+  Write-Host "                                 (Example: -vs Microsoft.CSharp or -vs CoreCLR.sln)"
   Write-Host ""
 
   Write-Host "Actions (defaults to -restore -build):"
@@ -79,7 +80,8 @@ function Get-Help() {
 
   Write-Host "Native build settings:"
   Write-Host "  -cmakeargs              User-settable additional arguments passed to CMake."
-  Write-Host "  -ninja                  Use Ninja instead of MSBuild to run the native build."
+  Write-Host "  -ninja                  Use Ninja to drive the native build. (default)"
+  Write-Host "  -msbuild                Use MSBuild to drive the native build. This is a no-op for Mono."
   Write-Host "  -pgoinstrument          Build the CLR with PGO instrumentation."
 
   Write-Host "Command-line arguments not listed above are passed through to MSBuild."
@@ -133,10 +135,10 @@ if ($vs) {
     if ($runtimeConfiguration) {
       $configToOpen = $runtimeConfiguration
     }
-    $vs = Split-Path $PSScriptRoot -Parent | Join-Path -ChildPath "artifacts\obj\coreclr" | Join-Path -ChildPath "windows.$archToOpen.$((Get-Culture).TextInfo.ToTitleCase($configToOpen))" | Join-Path -ChildPath "CoreCLR.sln"
+    $vs = Split-Path $PSScriptRoot -Parent | Join-Path -ChildPath "artifacts\obj\coreclr" | Join-Path -ChildPath "windows.$archToOpen.$((Get-Culture).TextInfo.ToTitleCase($configToOpen))" | Join-Path -ChildPath "ide" | Join-Path -ChildPath "CoreCLR.sln"
     if (-Not (Test-Path $vs)) {
       $repoRoot = Split-Path $PSScriptRoot -Parent
-      Invoke-Expression "& `"$repoRoot/src/coreclr/build-runtime.cmd`" -configureonly -$archToOpen -$configToOpen"
+      Invoke-Expression "& `"$repoRoot/src/coreclr/build-runtime.cmd`" -configureonly -$archToOpen -$configToOpen -msbuild"
       if ($lastExitCode -ne 0) {
         Write-Error "Failed to generate the CoreCLR solution file."
         exit 1
@@ -234,7 +236,9 @@ foreach ($argument in $PSBoundParameters.Keys)
     "properties"             { $arguments += " " + $properties }
     "verbosity"              { $arguments += " -$argument " + $($PSBoundParameters[$argument]) }
     "cmakeargs"              { $arguments += " /p:CMakeArgs=`"$($PSBoundParameters[$argument])`"" }
-    "ninja"                  { $arguments += " /p:Ninja=$($PSBoundParameters[$argument])" }
+    # The -ninja switch is a no-op since Ninja is the default generator on Windows.
+    "ninja"                  { }
+    "msbuild"                { $arguments += " /p:Ninja=false" }
     "pgoinstrument"          { $arguments += " /p:PgoInstrument=$($PSBoundParameters[$argument])"}
     # configuration and arch can be specified multiple times, so they should be no-ops here
     "configuration"          {}
@@ -276,4 +280,8 @@ if ($failedBuilds.Count -ne 0) {
     exit 1
 }
 
+if ($ninja) {
+    Write-Host "The -ninja option has no effect on Windows builds since the Ninja generator is the default generator."
+}
+
 exit 0

eng/native/configurecompiler.cmake

@@ -463,7 +463,11 @@ if (MSVC)
   # Compile options for targeting windows
 
   add_compile_options($<$<COMPILE_LANGUAGE:C,CXX>:/nologo>) # Suppress Startup Banner
-  add_compile_options($<$<COMPILE_LANGUAGE:C,CXX>:/W3>) # set warning level to 3
+  # /W3 is added by default by CMake, so remove it
+  string(REPLACE "/W3" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+  string(REPLACE "/W3" "" CMAKE_C_FLAGS "${CMAKE_C_FLAGS}")
+  # set default warning level to 3 but allow targets to override it.
+  add_compile_options($<$<COMPILE_LANGUAGE:C,CXX>:/W$<GENEX_EVAL:$<IF:$<BOOL:$<TARGET_PROPERTY:MSVC_WARNING_LEVEL>>,$<TARGET_PROPERTY:MSVC_WARNING_LEVEL>,3>>>)
   add_compile_options($<$<COMPILE_LANGUAGE:C,CXX>:/WX>) # treat warnings as errors
   add_compile_options($<$<COMPILE_LANGUAGE:C,CXX>:/Oi>) # enable intrinsics
   add_compile_options($<$<COMPILE_LANGUAGE:C,CXX>:/Oy->) # disable suppressing of the creation of frame pointers on the call stack for quicker function calls

eng/native/functions.cmake

@@ -466,34 +466,6 @@ function(add_executable_clr)
     _add_executable(${ARGV})
 endfunction()
 
-function(generate_module_index Target ModuleIndexFile)
-    if(CLR_CMAKE_HOST_WIN32)
-        set(scriptExt ".cmd")
-    else()
-        set(scriptExt ".sh")
-    endif()
-
-    set(index_timestamp ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${Target}_index.timestamp)
-
-    add_custom_command(
-        OUTPUT ${index_timestamp}
-        COMMAND ${CLR_ENG_NATIVE_DIR}/genmoduleindex${scriptExt} $<TARGET_FILE:${Target}> ${ModuleIndexFile}
-        COMMAND ${CMAKE_COMMAND} -E touch ${index_timestamp}
-        DEPENDS ${Target}
-        COMMENT "Generating ${Target} module index file -> ${ModuleIndexFile}"
-    )
-
-    set_source_files_properties(
-        ${ModuleIndexFile}
-        PROPERTIES GENERATED TRUE
-    )
-
-    add_custom_target(
-        ${Target}_module_index_header
-        DEPENDS ${index_timestamp}
-    )
-endfunction(generate_module_index)
-
 # add_linker_flag(Flag [Config1 Config2 ...])
 function(add_linker_flag Flag)
   if (ARGN STREQUAL "")

eng/pipelines/coreclr/templates/build-job.yml

@@ -132,12 +132,6 @@ jobs:
       - name: clrBuildPALTestsBuildArg
         value: '-paltests '
 
-    - name: ninjaArg
-      value: ''
-    - ${{ if eq(parameters.osGroup, 'windows') }}:
-      - name: ninjaArg
-        value: '-ninja'
-
     - name: pgoInstrumentArg
       value: ''
     - ${{ if eq(parameters.pgoType, 'PGO' )}}:
@@ -194,10 +188,10 @@ jobs:
 
     # Build CoreCLR Runtime
     - ${{ if ne(parameters.osGroup, 'windows') }}:
-      - script: $(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) $(crossArg) $(osArg) -ci $(compilerArg) $(clrBuildPALTestsBuildArg) $(ninjaArg) $(officialBuildIdArg) $(clrInterpreterBuildArg)
+      - script: $(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) $(crossArg) $(osArg) -ci $(compilerArg) $(clrBuildPALTestsBuildArg) $(officialBuildIdArg) $(clrInterpreterBuildArg)
         displayName: Build CoreCLR Runtime
     - ${{ if eq(parameters.osGroup, 'windows') }}:
-      - script: set __TestIntermediateDir=int&&$(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) -ci $(ninjaArg) $(enforcePgoArg) $(pgoInstrumentArg) $(officialBuildIdArg) $(clrInterpreterBuildArg)
+      - script: set __TestIntermediateDir=int&&$(Build.SourcesDirectory)/src/coreclr/build-runtime$(scriptExt) $(buildConfig) $(archType) -ci $(enforcePgoArg) $(pgoInstrumentArg) $(officialBuildIdArg) $(clrInterpreterBuildArg)
         displayName: Build CoreCLR Runtime
 
     - ${{ if in(parameters.osGroup, 'OSX', 'iOS', 'tvOS') }}:

eng/pipelines/global-build.yml

@@ -45,6 +45,21 @@ jobs:
       buildArgs: -c release -runtimeConfiguration debug
       timeoutInMinutes: 90
 
+#
+# Build with Release config and runtimeConfiguration with MSBuild generator
+#
+- template: /eng/pipelines/common/platform-matrix.yml
+  parameters:
+    jobTemplate: /eng/pipelines/common/global-build-job.yml
+    buildConfig: release
+    platforms:
+    - windows_x86
+    jobParameters:
+      testGroup: innerloop
+      nameSuffix: MSBuild_CMake
+      buildArgs: -c Release -msbuild
+      timeoutInMinutes: 90
+
 #
 # Build with Debug config and Release runtimeConfiguration
 #

src/coreclr/CMakeLists.txt

@@ -12,8 +12,6 @@ include(../../eng/native/configurepaths.cmake)
 include(${CLR_ENG_NATIVE_DIR}/configurecompiler.cmake)
 
 if(MSVC)
-  string(REPLACE "/EHsc" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
-  add_compile_options($<$<COMPILE_LANGUAGE:CXX>:/EHa>) # enable C++ EH (w/ SEH exceptions)
   set(CMAKE_CXX_STANDARD_LIBRARIES "") # do not link against standard win32 libs i.e. kernel32, uuid, user32, etc.
 endif (MSVC)
 
@@ -48,6 +46,21 @@ endif(CLR_CROSS_COMPONENTS_BUILD)
 #-------------------
 include(pgosupport.cmake)
 
+#---------------------------
+# Build the single file host
+#---------------------------
+if(CLR_CMAKE_BUILD_SUBSET_RUNTIME AND NOT CLR_CROSS_COMPONENTS_BUILD)
+  add_subdirectory(${CLR_SRC_NATIVE_DIR}/corehost/apphost/static Corehost.Static)
+endif()
+
+#-------------------------
+# Enable C++ EH with SEH
+#-------------------------
+if (MSVC)
+  string(REPLACE "/EHsc" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
+  add_compile_options($<$<COMPILE_LANGUAGE:CXX>:/EHa>) # enable C++ EH (w/ SEH exceptions)
+endif()
+
 #-------------------------------
 # Include libraries native shims
 #-------------------------------
@@ -130,10 +143,6 @@ if(FEATURE_STANDALONE_GC)
   endif (CLR_CMAKE_BUILD_SUBSET_RUNTIME)
 endif(FEATURE_STANDALONE_GC)
 
-if(CLR_CMAKE_BUILD_SUBSET_RUNTIME AND NOT CLR_CROSS_COMPONENTS_BUILD)
-  add_subdirectory(${CLR_SRC_NATIVE_DIR}/corehost/apphost/static Corehost.Static)
-endif()
-
 if (CLR_CMAKE_HOST_UNIX)
   include_directories("pal/inc")
   include_directories("pal/inc/rt")

src/coreclr/build-runtime.cmd

@@ -78,7 +78,7 @@ set __CrossArch2=
 set __CrossOS=0
 set __PgoOptDataPath=
 set __CMakeArgs=
-set __Ninja=0
+set __Ninja=1
 
 @REM CMD has a nasty habit of eating "=" on the argument list, so passing:
 @REM    -priority=1
@@ -152,7 +152,9 @@ if /i "%1" == "-skipnative"          (set __BuildNative=0&set processedArgs=!pro
 if /i "%1" == "-skipcrossarchnative" (set __SkipCrossArchNative=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "-skipgenerateversion" (set __SkipGenerateVersion=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "-skiprestoreoptdata"  (set __RestoreOptData=0&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
-if /i "%1" == "-ninja"               (set __Ninja=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+REM -ninja is a no-op option since Ninja is now the default generator on Windows.
+if /i "%1" == "-ninja"               (set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-msbuild"             (set __Ninja=0&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "-pgoinstrument"       (set __PgoInstrument=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "-enforcepgo"          (set __EnforcePgo=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "-nopgooptimize"       (set __PgoOptimize=0&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
@@ -284,7 +286,7 @@ set "__IntermediatesDir=%__RootBinDir%\obj\coreclr\%__TargetOS%.%__BuildArch%.%_
 set "__LogsDir=%__RootBinDir%\log\!__BuildType!"
 set "__MsbuildDebugLogsDir=%__LogsDir%\MsbuildDebugLogs"
 set "__ArtifactsIntermediatesDir=%__RepoRootDir%\artifacts\obj\coreclr\"
-if "%__Ninja%"=="1" (set "__IntermediatesDir=%__RootBinDir%\nmakeobj\%__TargetOS%.%__BuildArch%.%__BuildType%")
+if "%__Ninja%"=="0" (set "__IntermediatesDir=%__IntermediatesDir%\ide")
 set "__PackagesBinDir=%__BinDir%\.nuget"
 set "__CrossComponentBinDir=%__BinDir%"
 set "__CrossCompIntermediatesDir=%__IntermediatesDir%\crossgen"
@@ -707,6 +709,7 @@ REM ============================================================================
 
 echo %__MsgPrefix%Build succeeded.  Finished at %TIME%
 echo %__MsgPrefix%Product binaries are available at !__BinDir!
+
 exit /b 0
 
 REM =========================================================================================

src/coreclr/debug/runtimeinfo/CMakeLists.txt

@@ -6,11 +6,34 @@ set(RUNTIMEINFO_SOURCES
 
 add_library_clr(runtimeinfo STATIC ${RUNTIMEINFO_SOURCES})
 
-add_dependencies(runtimeinfo coreclr_module_index_header)
+function(generate_module_index Target ModuleIndexFile)
+    if(CLR_CMAKE_HOST_WIN32)
+        set(scriptExt ".cmd")
+    else()
+        set(scriptExt ".sh")
+    endif()
 
-if (NOT (CLR_CMAKE_TARGET_WIN32 AND (CLR_CMAKE_TARGET_ARCH_I386 OR CLR_CMAKE_TARGET_ARCH_ARM) AND CLR_CMAKE_HOST_ARCH_AMD64))
-    add_dependencies(runtimeinfo mscordaccore_module_index_header)
-    add_dependencies(runtimeinfo mscordbi_module_index_header)
+    add_custom_command(
+        OUTPUT ${ModuleIndexFile}
+        COMMAND ${CLR_ENG_NATIVE_DIR}/genmoduleindex${scriptExt} $<TARGET_FILE:${Target}> ${ModuleIndexFile}
+        DEPENDS ${Target}
+        COMMENT "Generating ${Target} module index file -> ${ModuleIndexFile}"
+    )
+
+    set_source_files_properties(
+        ${ModuleIndexFile}
+        PROPERTIES GENERATED TRUE
+    )
+
+    add_dependencies(runtimeinfo ${Target})
+    target_sources(runtimeinfo PRIVATE ${ModuleIndexFile})
+endfunction(generate_module_index)
+
+generate_module_index(coreclr ${CMAKE_CURRENT_BINARY_DIR}/runtimemoduleindex.h)
+if(NOT DEFINED CLR_CROSS_COMPONENTS_BUILD)
+    add_dependencies(runtimeinfo inject_debug_resources_coreclr)
+    generate_module_index(mscordaccore ${CMAKE_CURRENT_BINARY_DIR}/dacmoduleindex.h)
+    generate_module_index(mscordbi ${CMAKE_CURRENT_BINARY_DIR}/dbimoduleindex.h)
 endif()
 
 # publish runtimeinfo lib

src/coreclr/dlls/mscordac/CMakeLists.txt

@@ -192,12 +192,6 @@ endif(CLR_CMAKE_HOST_UNIX)
 
 target_link_libraries(mscordaccore PRIVATE ${COREDAC_LIBRARIES})
 
-# Create the DAC module index header file containing the DAC build id
-# for xplat and the timestamp/size on Windows.
-if(FEATURE_SINGLE_FILE_DIAGNOSTICS)
-    generate_module_index(mscordaccore ${GENERATED_INCLUDE_DIR}/dacmoduleindex.h)
-endif(FEATURE_SINGLE_FILE_DIAGNOSTICS)
-
 # add the install targets
 install_clr(TARGETS mscordaccore ADDITIONAL_DESTINATIONS sharedFramework)
 

src/coreclr/dlls/mscordbi/CMakeLists.txt

@@ -119,11 +119,5 @@ elseif(CLR_CMAKE_HOST_UNIX)
 
 endif(CLR_CMAKE_HOST_WIN32)
 
-# Create the DBI module index header file containing the DBI build id
-# for xplat and the timestamp/size on Windows.
-if(FEATURE_SINGLE_FILE_DIAGNOSTICS)
-    generate_module_index(mscordbi ${GENERATED_INCLUDE_DIR}/dbimoduleindex.h)
-endif(FEATURE_SINGLE_FILE_DIAGNOSTICS)
-
 # add the install targets
 install_clr(TARGETS mscordbi ADDITIONAL_DESTINATIONS sharedFramework)