Mismatch of cl.exe and STL version used in latest runner images #8259

dylanclark · 2023-09-08T22:02:46Z

Description

Our build pipelines in ADO are failing this week with the rollout of image 20230903.2.0 on windows-latest. The issue did not repro on 20230820.1.0.

We are seeing the failures while using VCPKG to build the cpprestsdk dependency. cpprestsdk is failing to build due to mismatched compiler and STL versions. It's possible it's the same root cause as #6091 where the MSVC install was broken. This does not repro on local dev machines.

Platforms affected

Azure DevOps
GitHub Actions - Standard Runners
GitHub Actions - Larger Runners

Runner images affected

Image version and build link

20230902.2.0. Build is on private ADO instance so can't link it.

Is it regression?

Regression. The issue did not repro on 20230820.1.0

Expected behavior

CMake picks the newest available versions of cl.exe and the STL headers.

Actual behavior

Cmake is picking an old version of the compiler and new version of STL headers.

cmake running under vcpkg is picking up an old compiler:

VERBOSE: Found C compiler at C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC*14.35.32215*\bin\HostX64\x64\cl.exe
VERBOSE: Found C++ compiler at C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC**\14.35.32215**bin\HostX64\x64\cl.exe

And cpprestsdk is referencing a different version of the STL headers:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC*14.37.32822*\include\yvals_core.h(851): error STL1001: Unexpected compiler version, expected MSVC 19.36 or newer.

VCPKG Logs

VERBOSE: Enabling Msvc compiler
VERBOSE: Calling RunVsDevCmd for x64
C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -latest -property installationPath
C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe -latest -property installationPath
cmd.exe /c "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64 && set
VERBOSE: Caching devenv environment variables to D:\a\1\s\build\vsDevCmdCache.x64.json
VERBOSE: Found C compiler at C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.35.32215\bin\HostX64\x64\cl.exe
VERBOSE: Found C++ compiler at C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.35.32215\bin\HostX64\x64\cl.exe
C:\vcpkg\vcpkg.exe version
vcpkg package management program version 2023-08-09-9990a4c9026811a312cb2af78bf77f3d9d288416

cpprestsdk build logs that contain the failure to build under vcpkg

Change Dir: 'C:/vcpkg/buildtrees/cpprestsdk/x64-windows-dbg'

Run Build Command(s): "C:/Program Files/Microsoft Visual Studio/2022/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/Ninja/ninja.exe" -v -v -j3 install
[1/25] C:\PROGRA~~1\MICROS~~2\2022\ENTERP~~1\VC\Tools\MSVC\1435~~1.322\bin\HostX64\x64\cl.exe /TP -DBROTLI_SHARED_COMPILATION -DCPPREST_EXCLUDE_WEBSOCKETS=1 -DUNICODE -DWIN32 -D_ASYNCRT_EXPORT -D_PPLX_EXPORT -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D_USRDLL -D_WIN32_WINNT=0x0600 -D_WINSOCK_DEPRECATED_NO_WARNINGS -Dcpprest_EXPORTS -IC:\vcpkg\buildtrees\cpprestsdk\src\ecb9e168c5-ec5efbc4a8.clean\Release\include -IC:\vcpkg\buildtrees\cpprestsdk\src\ecb9e168c5-ec5efbc4a8.clean\Release\src\pch -external:IC:\vcpkg\installed-x64-windows\x64-windows\include -external:W0 /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /MP /bigobj /permissive- /Yustdafx.h /FpC:/vcpkg/buildtrees/cpprestsdk/x64-windows-dbg/src/cpprest.pch /Zm120 /Ycstdafx.h /showIncludes /Fosrc\CMakeFiles\cpprest.dir\pch\stdafx.cpp.obj /Fdsrc\CMakeFiles\cpprest.dir\ /FS -c C:\vcpkg\buildtrees\cpprestsdk\src\ecb9e168c5-ec5efbc4a8.clean\Release\src\pch\stdafx.cpp
FAILED: src/CMakeFiles/cpprest.dir/pch/stdafx.cpp.obj
C:\PROGRA~~1\MICROS~~2\2022\ENTERP~~1\VC\Tools\MSVC\1435~~1.322\bin\HostX64\x64\cl.exe /TP -DBROTLI_SHARED_COMPILATION -DCPPREST_EXCLUDE_WEBSOCKETS=1 -DUNICODE -DWIN32 -D_ASYNCRT_EXPORT -D_PPLX_EXPORT -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D_USRDLL -D_WIN32_WINNT=0x0600 -D_WINSOCK_DEPRECATED_NO_WARNINGS -Dcpprest_EXPORTS -IC:\vcpkg\buildtrees\cpprestsdk\src\ecb9e168c5-ec5efbc4a8.clean\Release\include -IC:\vcpkg\buildtrees\cpprestsdk\src\ecb9e168c5-ec5efbc4a8.clean\Release\src\pch -external:IC:\vcpkg\installed-x64-windows\x64-windows\include -external:W0 /nologo /DWIN32 /D_WINDOWS /W3 /utf-8 /GR /EHsc /MP /D_DEBUG /MDd /Z7 /Ob0 /Od /RTC1 /MP /bigobj /permissive- /Yustdafx.h /FpC:/vcpkg/buildtrees/cpprestsdk/x64-windows-dbg/src/cpprest.pch /Zm120 /Ycstdafx.h /showIncludes /Fosrc\CMakeFiles\cpprest.dir\pch\stdafx.cpp.obj /Fdsrc\CMakeFiles\cpprest.dir\ /FS -c C:\vcpkg\buildtrees\cpprestsdk\src\ecb9e168c5-ec5efbc4a8.clean\Release\src\pch\stdafx.cpp
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\include\yvals_core.h(851): error STL1001: Unexpected compiler version, expected MSVC 19.36 or newer.
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\include\yvals_core.h(851): error C2338: static_assert failed: 'Error in C++ Standard Library usage.'
ninja: build stopped: subcommand failed.

Repro steps

The only repro we know of is to use VCPKG to build cpprestsdk on the latest image.

The latest GitHub runner image 20230903.2.0 shipped by GitHub for unknown reasons seemingly breaks MSVC debug builds. actions/runner-images#8259 and the fact that the previous image used a newer MSVC version than this one, suggest that the MSVC installation is currently broken. This sadly leads to the MultiThreadedDebug CI to be broken completely, making all builds fail. Try and workaround this issue by explicitly using the same toolset version from the previous runner-image in hope of it working.

shamil-mubarakshin · 2023-09-11T07:56:22Z

Hello @zero9178, image has both msvc 14.35 and 14.37 components installed, so versions can be specified/chosen while building the project.
And could you also try a following workaround:

- run: |
          Set-Location "C:\Program Files (x86)\Microsoft Visual Studio\Installer\"
          $InstallPath = "C:\Program Files\Microsoft Visual Studio\2022\Enterprise"
          $componentsToRemove= @(
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ARM"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ARM.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ARM64"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ARM64.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.x86.x64"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.x86.x64.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL.ARM"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL.ARM.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL.ARM64"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.ATL.ARM64.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC.ARM"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC.ARM.Spectre"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC.ARM64"
            "Microsoft.VisualStudio.Component.VC.14.35.17.5.MFC.ARM64.Spectre"
          )
          [string]$workloadArgs = $componentsToRemove | ForEach-Object {" --remove " +  $_}
          $Arguments = ('/c', "vs_installer.exe", 'modify', '--installPath', "`"$InstallPath`"",$workloadArgs, '--quiet', '--norestart', '--nocache')
          # should be run twice
          $process = Start-Process -FilePath cmd.exe -ArgumentList $Arguments -Wait -PassThru -WindowStyle Hidden
          $process = Start-Process -FilePath cmd.exe -ArgumentList $Arguments -Wait -PassThru -WindowStyle Hidden

zero9178 · 2023-09-11T08:03:35Z

Hi! Thank you for your reply. The surprising behaviour in my case was that it was using 14.35 by default instead of 14.37 when the previous version of the image used 14.36. That said, the precise version used doesn't really matter in my case and only threw me off during investigations. The issue ended up (not unexpectedely) in my application, so the CI failures while version dependentend, are not the fault of the version per-se.

I therefore also have no means really to test your workaround as my bug was unrelated. I suspect that your workaround may help @dylanclark though! The default version should probably be 14.37 given that it was 14.36 previously.

dylanclark · 2023-09-11T14:31:47Z

@shamil-mubarakshin Thank you, that workaround works in our build pipelines. However, it takes 10 minutes to run in our pipelines so it's not ideal. Hoping the next image release has a fix where the newest MSVC version is picked so we can remove this?

shamil-mubarakshin · 2023-09-12T11:13:25Z

@dylanclark, in case of msvc tools we are providing several versions of VS workflows. This approach allows version to be specified in build settings.
And yet, how VS determines which version should be set as default leaves me confused as well. Latest version is not picked up, no matter which installation order is chosen. I managed to find a VS issue under investigation with similar symptoms, but it has been this way for quite some time.
I will close this issue for now. Feel free to reach out in case of further concerns

toothache · 2023-10-24T13:47:12Z

MSVC version 14.35 is out of support. Should we still support it?

toothache · 2023-10-24T13:55:16Z

@dylanclark, in case of msvc tools we are providing several versions of VS workflows. This approach allows version to be specified in build settings. And yet, how VS determines which version should be set as default leaves me confused as well. Latest version is not picked up, no matter which installation order is chosen.

VS2022 is by default using v14.37 (v143) right now. The default props will be placed under C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\Microsoft.VCToolsVersion.default.props.

However, if you install v14.35 at the same time, an extra props file will be placed under C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\Microsoft.VCToolsVersion.v143.default.props. Since v143 is the default toolset version, this props file seems to redirect MSVC tool version to 14.35.

Per my experience, removing files C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\Microsoft.VCToolsVersion.v143.default.* can also mitigate the issue.

swt2c · 2023-10-30T23:14:55Z

Per my experience, removing files C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\Microsoft.VCToolsVersion.v143.default.* can also mitigate the issue.

Thank you for this. It worked for me also.

dylanclark added bug report needs triage labels Sep 8, 2023

zero9178 mentioned this issue Sep 9, 2023

[ci] Workaround GitHub Runner MSVC version Pylir/Pylir#51

Closed

shamil-mubarakshin added Area: C/C++ OS: Windows and removed needs triage labels Sep 11, 2023

shamil-mubarakshin self-assigned this Sep 11, 2023

shamil-mubarakshin closed this as completed Sep 12, 2023

toothache mentioned this issue Oct 25, 2023

Remove MSVC 14.35 in vs2022 which is out of support #8654

Closed

10 tasks

ihnorton mentioned this issue Mar 21, 2024

Use Ninja in Windows CI and explicitly set the VS toolset version. TileDB-Inc/TileDB#4759

Merged

Kolfering mentioned this issue Apr 7, 2024

[OpenAL-Soft]build failure for x64-windows-static-md on Github Actions microsoft/vcpkg#38003

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mismatch of cl.exe and STL version used in latest runner images #8259

Mismatch of cl.exe and STL version used in latest runner images #8259

dylanclark commented Sep 8, 2023 •

edited

shamil-mubarakshin commented Sep 11, 2023

zero9178 commented Sep 11, 2023 •

edited

dylanclark commented Sep 11, 2023 •

edited

shamil-mubarakshin commented Sep 12, 2023

toothache commented Oct 24, 2023 •

edited

toothache commented Oct 24, 2023 •

edited

swt2c commented Oct 30, 2023

Mismatch of cl.exe and STL version used in latest runner images #8259

Mismatch of cl.exe and STL version used in latest runner images #8259

Comments

dylanclark commented Sep 8, 2023 • edited

Description

Platforms affected

Runner images affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

VCPKG Logs

cpprestsdk build logs that contain the failure to build under vcpkg

Repro steps

shamil-mubarakshin commented Sep 11, 2023

zero9178 commented Sep 11, 2023 • edited

dylanclark commented Sep 11, 2023 • edited

shamil-mubarakshin commented Sep 12, 2023

toothache commented Oct 24, 2023 • edited

toothache commented Oct 24, 2023 • edited

swt2c commented Oct 30, 2023

dylanclark commented Sep 8, 2023 •

edited

zero9178 commented Sep 11, 2023 •

edited

dylanclark commented Sep 11, 2023 •

edited

toothache commented Oct 24, 2023 •

edited

toothache commented Oct 24, 2023 •

edited