Enable symlink to installed executables for win #3760

nanaceba · 2023-01-09T23:27:02Z

Summary of changes

Original issue:
Symlinks to installed generated executables are not working correctly on windows.

C:\>cd Snapchat\Dev\_nlo_\python\Scripts

C:\Snapchat\Dev\_nlo_\python\Scripts>conan_build_info.exe
usage: conan_build_info [-h] [--output OUTPUT] trace_path
conan_build_info: error: the following arguments are required: trace_path

WARN: Use 'conan_build_info --v2' to see the usage of the new recommended way to generate build info using lockfiles

C:\Snapchat\Dev\_nlo_\python\Scripts>cd ..

C:\Snapchat\Dev\_nlo_\python>cd ..

C:\Snapchat\Dev\_nlo_>cd bin

C:\Snapchat\Dev\_nlo_\bin>mklink conan_build_info.exe C:\Snapchat\Dev\_nlo_\python\Scripts\conan_build_info.exe
symbolic link created for conan_build_info.exe <<===>> C:\Snapchat\Dev\_nlo_\python\Scripts\conan_build_info.exe

C:\Snapchat\Dev\_nlo_\bin>conan_build_info.exe
Cannot open C:\Snapchat\Dev\_nlo_\bin\conan_build_info-script.py

We expect that this symlink should work correctly.

Solution:
In order to add correct symlink support for windows we are using windows api GetFinalPathNameByHandle that allows us to get symlink destination. If case if it is not symlink this function will just return the original file path. If it is symlink it will return symlink destination.

Changes:

Use GetFinalPathNameByHandle from winapi that resolve symlink/file path in launcher.c. This api supported from the Windows 6.1 as in original build tools.
Create CMake project that allows us to build with previous requirements set (CMAKE_SYSTEM_VERSION 6.1 CACHE TYPE INTERNAL FORCE) - this allows us to stay at the same API as before.
Modify build scripts to use CMake and latest Visual Studio MSBuild to build launcher executables for different arches.
Rebuild all windows wrappers using new build scripts.

Closes #3759

Pull Request Checklist

Changes have tests
News fragment added in changelog.d/.
(See documentation for details)

abravalheri · 2023-01-11T16:35:14Z

Hi @nanaceba, @eugene-sevostianov-sc thank you very much for the contribution.

I will probably defer the review of this PR to other maintainers that know better about this part of setuptools, but I think we need tests to be added to the PR.

This also seems to introduce a new piece in the build tool (cmake), which I am not entirely sure is desirable.

Quick comment, is it related to the entry-point executable wrappers?
(Unless you use the deprecated mechanism python setup.py install, those wrappers for executables are created by pip with the help of the distlib -- no longer created by setuptools)

jaraco

First, thanks. This looks like a solid contrib. Thorough work on revisiting this technique, which is very old (and maybe broken since I last checked).

abravalheri is correct. The script wrappers / launcher supplied by Setuptools is deprecated and would be removed except that it's relied upon by editable-installed packages (pip install -e).

I'd prefer to see these custom-built wrappers obviated entirely by pip-installed wrappers or replace these wrappers with distlib-backed launchers.

I'd be okay with this PR except that it's expanding the duplication found in the build scripts and adding more complex logic to them (creating directories, deleting existing directories, changing directories). If we're going to accept any more complexity to the scripts, I'd like to see them ported to Python and consolidated (avoiding repetition).

Before accepting, I'll want the binaries to be generated by one of the maintainers, so they're committed/signed by a maintainer.

abravalheri suggested adding test(s). Tests would be nice to capture the failure and prevent a regression. I'd be okay to skip tests given that this functionality is deprecated anyway.

Would you be interested in porting the build scripts to a Python script that does the same thing?

tools/msvc-build-launcher.cmd

tools/msvc-build-launcher-arm64.cmd

tools/msvc-build-launcher.cmd

nanaceba · 2023-01-16T21:26:11Z

@jaraco @abravalheri thank you guys for your comments!

To clarify, we build cpython per platform by ourselves to be more in control of how we distribute our environment and therefore by default it looks like it doesn't have wheel package, so the installation defaults to this old setuptools wrapper.
I didn't know that I might just need to install wheel package and symlinks are gonna work ok, but here we are :)

Basically right now any package we install using legacy installation in our environment

PS C:\Snapchat\Dev\test> python -m pip install .
Processing c:\snapchat\dev\test
  Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for conan-updater-wrapper, since package 'wheel' is not installed.
Installing collected packages: conan-updater-wrapper
  Attempting uninstall: conan-updater-wrapper
    Found existing installation: conan-updater-wrapper 0.0.0
    Uninstalling conan-updater-wrapper-0.0.0:
      Successfully uninstalled conan-updater-wrapper-0.0.0
  Running setup.py install for conan-updater-wrapper ... done
Successfully installed conan-updater-wrapper-0.0.0
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the 'C:\Snapchat\Dev\_nlo_\python\python.exe -m pip install --upgrade pip' command.

I guess its something that needs to be fixed upstream in pip to just get rid of it. But I think maybe someone else would encounter that.

I rewrote the build script to python and tested it. Feel free to rebuild and sign executables using the new build script.
Also I added a test for symlink case.

jaraco · 2023-01-17T00:15:54Z

This looks amazing. Thanks. I'll re-generate the binaries and make sure they produce no diffs (or accept the diffs they produce), but otherwise, this is good to go.

jaraco · 2023-01-17T00:41:04Z

Oh, spoke too soon. Looks like from the test failures, this change breaks .pyw scripts.

nanaceba · 2023-01-17T00:47:31Z

looking into failed tests

tools/build_launchers.py

jaraco · 2023-01-17T01:23:15Z

After installing cmake from cmake.org and running in a Visual Studio command prompt, I got some success, but still ran into a failure. It's quite possible the failure is due to the fact that I'm building on an arm64 Windows machine.

C:\Users\jaraco\code\pypa\setuptools>py tools\build_launchers.py
CMake Warning (dev) at CMakeLists.txt:2 (set):
  implicitly converting 'TYPE' to 'STRING' type.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 6.1.
-- The C compiler identification is MSVC 19.32.31332.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.32.31326/bin/HostARM64/x86/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/jaraco/code/pypa/setuptools/build-arena
Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/16/2023 8:02:42 PM.
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" on node 1 (build target(s)).
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (1) is building "C:\Users\jaraco\code\pypa\
setuptools\build-arena\ZERO_CHECK.vcxproj" (2) on node 1 (default targets).
PrepareForBuild:
  Creating directory "Win32\Release\ZERO_CHECK\".
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK
.vcxproj]
  Creating directory "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\".
InitializeBuildStatus:
  Creating "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Checking Build System
FinalizeBuildStatus:
  Deleting file "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild".
  Touching "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default targets).

PrepareForBuild:
  Creating directory "launcher.dir\Release\".
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.v
cxproj]
  Creating directory "launcher.dir\Release\launcher.tlog\".
InitializeBuildStatus:
  Creating "launcher.dir\Release\launcher.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Building Custom Rule C:/Users/jaraco/code/pypa/setuptools/launcher/CMakeLists.txt
ClCompile:
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\HostArm64\x86\CL.exe /c /nologo
   /W1 /WX- /diagnostics:column /O2 /Ob2 /Oy- /D _MBCS /D WIN32 /D _WINDOWS /D NDEBUG /D GUI /D WIN32_LEAN_AND_MEAN /D
  "CMAKE_INTDIR=\"Release\"" /Gm- /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"launcher.dir\Release\\" /
  Fd"launcher.dir\Release\vc143.pdb" /external:W1 /Gd /TC /analyze- /errorReport:queue C:\Users\jaraco\code\pypa\setupt
  ools\launcher.c
  launcher.c
MakeDirsForLink:
  Creating directory "C:\Users\jaraco\code\pypa\setuptools\build-arena\Release".
Link:
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\HostArm64\x86\link.exe /ERRORRE
  PORT:QUEUE /OUT:"C:\Users\jaraco\code\pypa\setuptools\setuptools\cli-32.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib use
  r32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIF
  ESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:"C:/Users/jaraco/code/pypa/setuptools/build-arena/Re
  lease/launcher.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/Users/jaraco/code/pypa/setuptools/
  build-arena/Release/launcher.lib" /MACHINE:X86 /SAFESEH  /machine:X86 launcher.dir\Release\launcher.obj
  launcher.vcxproj -> C:\Users\jaraco\code\pypa\setuptools\setuptools\cli-32.exe
FinalizeBuildStatus:
  Deleting file "launcher.dir\Release\launcher.tlog\unsuccessfulbuild".
  Touching "launcher.dir\Release\launcher.tlog\launcher.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target(s)).


Build succeeded.

"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
"C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default target) (2) ->
(PrepareForBuild target) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHE
CK.vcxproj]


"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher
.vcxproj]

    2 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.95
Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/16/2023 8:02:43 PM.
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" on node 1 (build target(s)).
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (1) is building "C:\Users\jaraco\code\pypa\
setuptools\build-arena\ZERO_CHECK.vcxproj" (2) on node 1 (default targets).
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK
.vcxproj]
InitializeBuildStatus:
  Creating "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  All outputs are up-to-date.
FinalizeBuildStatus:
  Deleting file "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild".
  Touching "Win32\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default targets).

C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.v
cxproj]
InitializeBuildStatus:
  Creating "launcher.dir\Release\launcher.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  All outputs are up-to-date.
ClCompile:
  All outputs are up-to-date.
Link:
  All outputs are up-to-date.
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\HostArm64\x86\link.exe /ERRORRE
  PORT:QUEUE /OUT:"C:\Users\jaraco\code\pypa\setuptools\setuptools\gui-32.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib use
  r32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIF
  ESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:"C:/Users/jaraco/code/pypa/setuptools/build-arena/Re
  lease/launcher.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/Users/jaraco/code/pypa/setuptools/
  build-arena/Release/launcher.lib" /MACHINE:X86 /SAFESEH  /machine:X86 launcher.dir\Release\launcher.obj
  launcher.vcxproj -> C:\Users\jaraco\code\pypa\setuptools\setuptools\gui-32.exe
FinalizeBuildStatus:
  Deleting file "launcher.dir\Release\launcher.tlog\unsuccessfulbuild".
  Touching "launcher.dir\Release\launcher.tlog\launcher.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target(s)).


Build succeeded.

"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
"C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default target) (2) ->
(PrepareForBuild target) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHE
CK.vcxproj]


"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher
.vcxproj]

    2 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.45
CMake Warning (dev) at CMakeLists.txt:2 (set):
  implicitly converting 'TYPE' to 'STRING' type.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 6.1.
-- The C compiler identification is MSVC 19.32.31332.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.32.31326/bin/HostARM64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/jaraco/code/pypa/setuptools/build-arena
Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/16/2023 8:02:45 PM.
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" on node 1 (build target(s)).
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (1) is building "C:\Users\jaraco\code\pypa\
setuptools\build-arena\ZERO_CHECK.vcxproj" (2) on node 1 (default targets).
PrepareForBuild:
  Creating directory "x64\Release\ZERO_CHECK\".
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK
.vcxproj]
  Creating directory "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\".
InitializeBuildStatus:
  Creating "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Checking Build System
FinalizeBuildStatus:
  Deleting file "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild".
  Touching "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default targets).

PrepareForBuild:
  Creating directory "launcher.dir\Release\".
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.v
cxproj]
  Creating directory "launcher.dir\Release\launcher.tlog\".
InitializeBuildStatus:
  Creating "launcher.dir\Release\launcher.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Building Custom Rule C:/Users/jaraco/code/pypa/setuptools/launcher/CMakeLists.txt
ClCompile:
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\HostArm64\x64\CL.exe /c /nologo
   /W1 /WX- /diagnostics:column /O2 /Ob2 /D _MBCS /D WIN32 /D _WINDOWS /D NDEBUG /D GUI /D WIN32_LEAN_AND_MEAN /D "CMAK
  E_INTDIR=\"Release\"" /Gm- /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"launcher.dir\Release\\" /Fd"la
  uncher.dir\Release\vc143.pdb" /external:W1 /Gd /TC /errorReport:queue C:\Users\jaraco\code\pypa\setuptools\launcher.c
  launcher.c
MakeDirsForLink:
  Creating directory "C:\Users\jaraco\code\pypa\setuptools\build-arena\Release".
Link:
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\HostArm64\x64\link.exe /ERRORRE
  PORT:QUEUE /OUT:"C:\Users\jaraco\code\pypa\setuptools\setuptools\cli-64.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib use
  r32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIF
  ESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:"C:/Users/jaraco/code/pypa/setuptools/build-arena/Re
  lease/launcher.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/Users/jaraco/code/pypa/setuptools/
  build-arena/Release/launcher.lib" /MACHINE:X64  /machine:x64 launcher.dir\Release\launcher.obj
  launcher.vcxproj -> C:\Users\jaraco\code\pypa\setuptools\setuptools\cli-64.exe
FinalizeBuildStatus:
  Deleting file "launcher.dir\Release\launcher.tlog\unsuccessfulbuild".
  Touching "launcher.dir\Release\launcher.tlog\launcher.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target(s)).


Build succeeded.

"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
"C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default target) (2) ->
(PrepareForBuild target) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHE
CK.vcxproj]


"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher
.vcxproj]

    2 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.78
Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/16/2023 8:02:46 PM.
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" on node 1 (build target(s)).
Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (1) is building "C:\Users\jaraco\code\pypa\
setuptools\build-arena\ZERO_CHECK.vcxproj" (2) on node 1 (default targets).
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK
.vcxproj]
InitializeBuildStatus:
  Creating "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  All outputs are up-to-date.
FinalizeBuildStatus:
  Deleting file "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild".
  Touching "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default targets).

C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5): wa
rning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is re
quired to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.v
cxproj]
InitializeBuildStatus:
  Creating "launcher.dir\Release\launcher.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  All outputs are up-to-date.
ClCompile:
  All outputs are up-to-date.
Link:
  All outputs are up-to-date.
  C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.32.31326\bin\HostArm64\x64\link.exe /ERRORRE
  PORT:QUEUE /OUT:"C:\Users\jaraco\code\pypa\setuptools\setuptools\gui-64.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib use
  r32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIF
  ESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:"C:/Users/jaraco/code/pypa/setuptools/build-arena/Re
  lease/launcher.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/Users/jaraco/code/pypa/setuptools/
  build-arena/Release/launcher.lib" /MACHINE:X64  /machine:x64 launcher.dir\Release\launcher.obj
  launcher.vcxproj -> C:\Users\jaraco\code\pypa\setuptools\setuptools\gui-64.exe
FinalizeBuildStatus:
  Deleting file "launcher.dir\Release\launcher.tlog\unsuccessfulbuild".
  Touching "launcher.dir\Release\launcher.tlog\launcher.lastbuildstate".
Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target(s)).


Build succeeded.

"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
"C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHECK.vcxproj" (default target) (2) ->
(PrepareForBuild target) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\ZERO_CHE
CK.vcxproj]


"C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher.vcxproj" (build target) (1) ->
  C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(500,5):
warning MSB8004: Output Directory does not end with a trailing slash.  This build instance will add the slash as it is
required to allow proper evaluation of the Output Directory. [C:\Users\jaraco\code\pypa\setuptools\build-arena\launcher
.vcxproj]

    2 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.43
CMake Warning (dev) at CMakeLists.txt:2 (set):
  implicitly converting 'TYPE' to 'STRING' type.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Selecting Windows SDK version 10.0.19041.0 to target Windows 6.1.
CMake Error at CMakeLists.txt:3 (project):
  Failed to run MSBuild command:

    C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/arm64/MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 1/16/2023 8:02:47 PM.
    Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\CMakeFiles\3.25.1\VCTargetsPath.vcxproj" on node 1 (default targets).
    C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\arm64\Microsoft.Common.CurrentVersion.targets(824,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='arm64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\Users\jaraco\code\pypa\setuptools\build-arena\CMakeFiles\3.25.1\VCTargetsPath.vcxproj]
    Done Building Project "C:\Users\jaraco\code\pypa\setuptools\build-arena\CMakeFiles\3.25.1\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.

    "C:\Users\jaraco\code\pypa\setuptools\build-arena\CMakeFiles\3.25.1\VCTargetsPath.vcxproj" (default target) (1) ->
    (_CheckForInvalidConfigurationAndPlatform target) ->
      C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\arm64\Microsoft.Common.CurrentVersion.targets(824,5): error : The BaseOutputPath/OutputPath property is not set for project 'VCTargetsPath.vcxproj'.  Please check to make sure that you have specified a valid combination of Configuration and Platform for this project.  Configuration='Debug'  Platform='arm64'.  You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project. [C:\Users\jaraco\code\pypa\setuptools\build-arena\CMakeFiles\3.25.1\VCTargetsPath.vcxproj]

        0 Warning(s)
        1 Error(s)

    Time Elapsed 00:00:00.10


  Exit code: 1



-- Configuring incomplete, errors occurred!
See also "C:/Users/jaraco/code/pypa/setuptools/build-arena/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "C:\Users\jaraco\code\pypa\setuptools\tools\build_launchers.py", line 76, in <module>
    main()
    ^^^^^^
  File "C:\Users\jaraco\code\pypa\setuptools\tools\build_launchers.py", line 57, in main
    generate_cmake_project(build_arena, LAUNCHER_CMAKE_PROJECT.resolve(), platform)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jaraco\code\pypa\setuptools\tools\build_launchers.py", line 41, in generate_cmake_project
    subprocess.check_call(f'cmake -G "{VISUAL_STUDIO_VERSION}" -A "{platform}"'
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311-arm64\Lib\subprocess.py", line 402, in check_call
    raise CalledProcessError(retcode, cmd)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
subprocess.CalledProcessError: Command 'cmake -G "Visual Studio 17 2022" -A "arm64" C:\Users\jaraco\code\pypa\setuptools\launcher' returned non-zero exit status 1.

Can you investigate the output and confirm my suspicion that the issue is caused by Windows on ARM, or is there someithing else here that needs addressing?

nanaceba · 2023-01-17T03:32:19Z

@jaraco just deployed the fix to binaries and explanation to ARM problem with visual studio. It's added to the build file docs.
In short, to build ARM on windows you need install additional components.
Also I found a bug why tests were failing - MSBuild doesn't support passing compile definitions, so we passing GUI value to cmake when it generates the project. Tests passed locally for me.
Added a workaround to locate MSBuild binary without running any additional shell.

ARM building:

From visual studio installer:
Visual Studio -> Modify -> Individual Components

List of components to install to compile ARM:
C++ Universal Windows Platform Support for v143 build Tools (ARM64)
MSVC v143 - VS 2022 C++ ARM64 build tools (latest)
MSVC v143 - VS 2022 C++ ARM64 Spectre-mitigated libs (latest)
C++ ATL for latest v143 build tools (ARM64)

nanaceba · 2023-01-17T04:02:01Z

@jaraco can you ensure that you have https://devblogs.microsoft.com/dotnet/announcing-dotnet-framework-481/ 4.81 dotnet framework installed? Seems like your issue related to the CMake and arm64. https://gitlab.kitware.com/cmake/cmake/-/merge_requests/7511 seems like that issue was fixed if you are on the latest cmake installation.
Also are you on windows 10 or 11?
I hope visual studio arm64 build tools should help.

jaraco · 2023-05-19T19:40:10Z

I'm unsure what version of .Net framework I have installed (there's no indication of the version in my installed programs). Since the last time I looked at this issue, though, I've reinstalled my Windows machine (Windows 11 again), so I have a very late version of Visual Studio, almost certainly with the latest .Net framework. I'm adding the indicated features of Visual Studio and I'll retry today.

…turally to avoid reliance on shell.

nanaceba · 2023-05-23T16:39:11Z

@jaraco thank you!

jaraco · 2023-05-24T01:40:07Z

And likewise, thank you. Your work here has helped simplify the codebase and sustain the maintenance of these scripts. As much as I'd like to see them be replaced by something else, I feel much better about maintaining them than I ever have.

nanaceba · 2024-02-03T00:32:26Z

@jaraco @abravalheri just had an incident - we actually migrated to the wheel but the case still living - if you try to install python package as editable with -m pip install --editable . it still gonna use this installation. So it seems not everything migrated in a wheel :)
Updating to latest setup tools fixed the issue, and I came back to report to you that these use cases still exist even with a wheel!

eugene-sevostianov-sc added 2 commits January 9, 2023 14:50

enable symlink for win

4bf6f12

add changelog info

02100ea

jaraco requested changes Jan 16, 2023

View reviewed changes

tools/msvc-build-launcher.cmd Outdated Show resolved Hide resolved

tools/msvc-build-launcher-arm64.cmd Outdated Show resolved Hide resolved

tools/msvc-build-launcher.cmd Outdated Show resolved Hide resolved

introduce proper build script on python

f0dc7d8

nanaceba requested a review from jaraco January 16, 2023 21:30

eugene-sevostianov-sc added 2 commits January 16, 2023 13:39

nits

57cb20a

nits

3c17710

jaraco reviewed Jan 17, 2023

View reviewed changes

tools/build_launchers.py Outdated Show resolved Hide resolved

jaraco reviewed Jan 17, 2023

View reviewed changes

tools/build_launchers.py Outdated Show resolved Hide resolved

fix issues

182f685

nanaceba requested a review from jaraco January 17, 2023 03:43

update the way we pass definitions to cmake

77b3ec5

jaraco added 9 commits May 19, 2023 19:56

⚫ Fade to black.

3db9af4

Return discovered msbuild_path value.

cd832f5

Rewrite get_msbuild to unconditionally use vswhere. Resolve values na…

7f9cef7

…turally to avoid reliance on shell.

Locate CMake deterministically using registry.

0981930

Move attribute docstring to after the attribute.

8cb3332

Update the module docstring.

774e876

Re-use global get_* functions.

b11a9ee

Convert subprocess calls to use explicit parameters.

8f5dbeb

Extract function for resolve_platform.

11877de

jaraco added 5 commits May 19, 2023 21:23

Extract function for doing the build.

605aa25

Generate a temporary directory for the build arena.

eabe37a

Use itertools.product instead of nested loops.

bebd6bf

Rebuilt the executables.

ed1b6fb

Merge branch 'main' into es/enable_symlinks_for_windows

50c6e32

jaraco approved these changes May 20, 2023

View reviewed changes

jaraco merged commit e4c8cdb into pypa:main May 20, 2023
23 checks passed

scdub mentioned this pull request Jun 16, 2023

[BUG] PE wrappers missing Address Space Layout Randomization (ASLR) and Data Execution Prevention (DEP) #3029

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable symlink to installed executables for win #3760

Enable symlink to installed executables for win #3760

nanaceba commented Jan 9, 2023 •

edited

abravalheri commented Jan 11, 2023 •

edited

jaraco left a comment

nanaceba commented Jan 16, 2023 •

edited

jaraco commented Jan 17, 2023

jaraco commented Jan 17, 2023

nanaceba commented Jan 17, 2023

jaraco commented Jan 17, 2023 •

edited

nanaceba commented Jan 17, 2023 •

edited

nanaceba commented Jan 17, 2023 •

edited

jaraco commented May 19, 2023

nanaceba commented May 23, 2023

jaraco commented May 24, 2023

nanaceba commented Feb 3, 2024

Enable symlink to installed executables for win #3760

Enable symlink to installed executables for win #3760

Conversation

nanaceba commented Jan 9, 2023 • edited

Summary of changes

Pull Request Checklist

abravalheri commented Jan 11, 2023 • edited

jaraco left a comment

Choose a reason for hiding this comment

nanaceba commented Jan 16, 2023 • edited

jaraco commented Jan 17, 2023

jaraco commented Jan 17, 2023

nanaceba commented Jan 17, 2023

jaraco commented Jan 17, 2023 • edited

nanaceba commented Jan 17, 2023 • edited

nanaceba commented Jan 17, 2023 • edited

jaraco commented May 19, 2023

nanaceba commented May 23, 2023

jaraco commented May 24, 2023

nanaceba commented Feb 3, 2024

nanaceba commented Jan 9, 2023 •

edited

abravalheri commented Jan 11, 2023 •

edited

nanaceba commented Jan 16, 2023 •

edited

jaraco commented Jan 17, 2023 •

edited

nanaceba commented Jan 17, 2023 •

edited

nanaceba commented Jan 17, 2023 •

edited