[python3] Improve support for embedding and add tools #13556

Hoikas · 2020-09-16T04:16:41Z

This pull request improves the downstream experience when working with the python3 port by building a functioning interpreter with support for pip and improving support when cmake's FindPython3 finder is used. Previously, when using FindPython3, if both the DEVELOPMENT and INTERPRETER components were requested, you would have to have a local python install matching the vcpkg python. Further, on Windows, this module preferred registry python over any slightly misconfigured python artifacts. Fixing these issues allows us to simplify ports whose cmake build systems need the python3 development libraries.

Anyone who intends to do further work on the python3 port after this might consider fixing vcpkg_find_acquire_program(PYTHON3) to use the interpreter installed by this port.

Note that on Windows a patch is applied in static python interpreters to disable extension modules-due to the way DLLs function, it is not possible to load them without crashing the interpreter. Further, support for the static CRT has been removed both to avoid surprises in that regard and due to the nature of PyObjects.

What does your PR fix?
This should address [python3] cmake FindPython3 Development component #9026 and [python3] Install python executable from port #10903.
Which triplets are supported/not supported? Have you updated the CI baseline?
Support for x64-windows-static has been dropped.
Does your PR follow the maintainer guide?
I think so 😄

PhoebeHui · 2020-09-17T10:37:10Z

Thanks for the PR!

@ras0219-msft, could you help further review?

JackBoosY · 2020-09-18T02:23:14Z

ports/cppcms/portfile.cmake

@@ -15,7 +15,7 @@ get_filename_component(PYTHON2_DIR ${PYTHON2} DIRECTORY)
 vcpkg_configure_cmake(
    SOURCE_PATH ${SOURCE_PATH}
    PREFER_NINJA
-    OPTIONS -DCMAKE_PROGRAM_PATH=${PYTHON2_DIR} -DUSE_WINDOWS6_API=ON
+    OPTIONS -DPYTHON:FILEPATH=${PYTHON2} -DCMAKE_PROGRAM_PATH=${PYTHON2_DIR} -DUSE_WINDOWS6_API=ON


I like this writing. it's tiny and beautiful!

ras0219-msft · 2020-09-24T19:22:33Z

Thanks for this PR!

Python is a very tricky subject in vcpkg, because there are three different use cases:

Libraries want to run python code during their build -- this means they need python for the host machine. This case is when ports should be using vcpkg_find_acquire_program(PYTHON3).
Libraries want to link against python.dll/.a/.lib because they expect the consumer to be a c++ program that embeds python -- this means they need python for the target machine. This is covered by the python3 port.
Libraries want to link against python.dll/.so because they are authoring extension modules to be used with an external python. This is the stickier situation, because it's possible that the ABI of the external python does not match the ABI of the internal python. However, I've been lead to believe that on Windows at least, python3 should generally be ABI compatible and so in this case it's acceptable to use the python3 port for building. I think same situation may apply for Linux, but I'm less certain of this platform.

Framed by the above three circumstances, could you re-explain what this PR is intended to accomplish? It's important to keep these situations separate in order to enable cross compilation.

Hoikas · 2020-09-24T20:13:30Z

Agreed, Python is definitely very tricky. Given the three use case you pointed out, this pull request was designed to address issues I've experienced in projects I contribute to around use case 2 and, to some extent, use case 1.

When using vcpkg's python with any of cmake's FindPython modules on Windows, they first go to the registry. The find modules introduced in CMake 3.12 (FindPython, FindPython2, FindPython3) add components such as Development and Interpreter. When a downstream user of vcpkg python is both embedding and using python as part of their build process eg by find_package(Python3 REQUIRED COMPONENTS Interpreter Development), the find will fail if the user does not have an installed system Python that is ABI compatible with the vcpkg python, assuming that the user correctly manually specifies the vcpkg artifacts.

Additionally, specifying the vcpkg artifacts is tricky with these finders, especially when using a multi-config generator (eg Visual Studio). A common complaint from our users is that the debug library is not properly found, leading to either surprise link errors with python38_d.lib not being found or with FindPython deciding to use some other non-vcpkg python. Overall, this leads to an unacceptable level of surprises and GOTCHAS, IMO. This pull request fixes that by building the python interpreter and ensuring artifacts are properly specified for all five cmake find python variants: FindPythonLibs, FindPythonInterp, FindPython, FindPython2, and FindPython3.

In the case of embedding, having python without its standard library is largely pointless. In fact, in Python 3, python cannot be correctly embedded without the standard library. It imports the encodings module on startup and fails if the standard library is unavailable. CMake 3.12's new find modules return the location of the Python standard library -- which is mostly a set of python scripts. However, to do so, it must run the interpreter itself. This brings us back to the previous point. Further, the standard library requires several internal extension modules to be built. This PR causes those extension modules to be built--they were not previously.

I built a project to test what I expect to work on Windows, macOS, and Linux here. (I disabled Windows CI in that repository in favor of local testing.)

Several ports fall into multiple use cases in that they use python3 in their build process and either embed or extend it. In those cases, we find fragile artifact specifications for the python development libraries in the portfile.cmake and a call to download the python interpreter. With a properly functioning python.exe built by vcpkg, these use cases can be simplified fairly significantly... Unless the upstream library rolls its own python finder (eg pybind11). If, on the other hand, the port only needs python as part of its build process, use case 1 remains unaffected.

I did address one issue related to case 3 while working on the PR. Python is ABI stable at the minor release level (eg, 3.8.3 is compatible with 3.8.5 but not with 3.9.0) unless using the Py_LIMITED_API, which is stable across all of python 3. vcpkg neither supports this option presently nor with this PR. One undocumented sticking point I found with extension modules is related to the library linkage. On Windows, extension modules can only function against a shared python library. Attempting to link an extension module against the static python library will crash the interpreter when the end user imports said module. This limitation is not in present on unix-like platforms. Previously, on Windows, this was a silent GOTCHA. I did not outright remove support for static triplets because the static library is incredibly helpful when embedding python in use case 2.

This allows overriding the name of the install target, allowing specification of something other than `make install`, eg `make altinstall`.

extension.

- Many deprecated functions were removed. - A working python interpreter is now installed under tools. - The import library is now properly frozen into the interpreter.

This allows us to prevent CMake going to the registry or otherwise exploding when using: - FindPython - FindPython3 - FindPythonInterp - FindPythonLibs

possible.

rejected.

The upstream port supports python3 now, but a new release is not yet available.

It seems pybind11 has its own python find module. The current upstream head allows using CMake's new FindPython module with the definition `-DPYBIND11_FINDPYTHON`, but there has been no upstream release with this feature. In the meantime, we correctly find the vcpkg python3 interpreter and libraries.

This fixes an observed build failure on x86-windows when the vcpkg expat port is installed before the python3 port is built.

Hoikas · 2020-09-25T09:04:40Z

This has been rebased to merge cleanly with master.

JackBoosY · 2020-09-25T09:39:26Z

blocked by #13722.

Hoikas · 2020-09-25T16:24:06Z

I am unable to reproduce the CI failures with irrilicht and qt5-location locally.

JackBoosY · 2020-09-27T07:26:09Z

blocked by #13765.

Hoikas · 2020-10-01T20:51:05Z

CI issues fixed in #13812 and #13830.

JackBoosY · 2020-10-09T09:14:17Z

osg regression will be fixed in #13942.

Hoikas · 2020-10-12T21:00:27Z

After re-reading the comment by @ras0219-msft, I have reworked some of the commits in this pull request to properly support use case 1 by not assuming that the python interpreter build by vcpkg may be used on the host machine.

scripts/ci.baseline.txt

Hoikas · 2020-11-03T20:58:47Z

Before I take the time to resolve the merge conflicts, would it help the review process if this were split into multiple pull requests? It's been a little over a month since any substantial input was given.

JackBoosY · 2020-11-04T02:10:24Z

@Hoikas yes, that's better.

Hoikas force-pushed the py3 branch from 134cccc to b4a9be1 Compare September 16, 2020 04:29

JackBoosY assigned PhoebeHui Sep 16, 2020

Hoikas marked this pull request as ready for review September 17, 2020 02:25

PhoebeHui assigned JackBoosY Sep 17, 2020

PhoebeHui added the category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist label Sep 17, 2020

PhoebeHui assigned ras0219-msft Sep 17, 2020

JackBoosY reviewed Sep 18, 2020

View reviewed changes

Hoikas mentioned this pull request Sep 25, 2020

Python3 offer tool feature #13727

Closed

Hoikas added 10 commits September 25, 2020 05:04

[scripts/cmake/vcpkg_install_make] Add INSTALL_TARGET parameter.

297f582

This allows overriding the name of the install target, allowing specification of something other than `make install`, eg `make altinstall`.

[scripts/cmake/vcpkg_copy_tool_dependencies] Add *.pyd as a valid tool

2d096ed

extension.

[python3] Refactor port for modernization and tools.

545932d

- Many deprecated functions were removed. - A working python interpreter is now installed under tools. - The import library is now properly frozen into the interpreter.

[python3] Add vcpkg-cmake-wrapper.cmake

fbc3fff

This allows us to prevent CMake going to the registry or otherwise exploding when using: - FindPython - FindPython3 - FindPythonInterp - FindPythonLibs

[many ports] Remove manual python3 artifact specification where

b93303f

possible.

[python3] Bump to 3.8.5.

491795b

[python3] Format the manifest -- otherwise CI is incomprehensibly

84b4ecb

rejected.

[cppcms] Ensure python2 is used by the build process.

46f09bf

The upstream port supports python3 now, but a new release is not yet available.

[python3] Use vcpkg expat port instead of vendored code.

5d9fbb7

This fixes an observed build failure on x86-windows when the vcpkg expat port is installed before the python3 port is built.

Hoikas force-pushed the py3 branch from bdde8df to 5d9fbb7 Compare September 25, 2020 09:04

JackBoosY and others added 2 commits September 27, 2020 00:27

Merge branch 'master' of https://github.com/microsoft/vcpkg into py3

5a0632a

Merge remote-tracking branch 'origin/master' into py3

801b49c

Hoikas mentioned this pull request Oct 1, 2020

Add a vcpkg triplet file and update README to describe its use. H-uru/libhsplasma#187

Merged

Merge remote-tracking branch 'origin/master' into py3

156771c

Hoikas mentioned this pull request Oct 9, 2020

[vcpkg_find_acquire_program] Fix SWIG on non-Windows platforms. #13953

Merged

Hoikas force-pushed the py3 branch from 156771c to ecd48e9 Compare October 12, 2020 20:57

Hoikas force-pushed the py3 branch from ecd48e9 to 1912169 Compare October 12, 2020 23:36

JackBoosY suggested changes Oct 13, 2020

View reviewed changes

scripts/ci.baseline.txt Show resolved Hide resolved

JackBoosY added requires:author-response and removed requires:author-response labels Oct 13, 2020

Hoikas mentioned this pull request Oct 15, 2020

[openimageio] Update to version 2.2.7.0 #13966

Closed

Hoikas mentioned this pull request Dec 2, 2020

[python3] Build interpreter #14891

Merged

This was referenced Dec 19, 2020

[python3] Add vcpkg-cmake-wrapper. #15220

Closed

[python3] Add vcpkg-cmake-wrapper. #15221

Merged

vicroms closed this in #15221 Jan 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python3] Improve support for embedding and add tools #13556

[python3] Improve support for embedding and add tools #13556

Hoikas commented Sep 16, 2020 •

edited

Loading

PhoebeHui commented Sep 17, 2020

JackBoosY Sep 18, 2020

ras0219-msft commented Sep 24, 2020 •

edited

Loading

Hoikas commented Sep 24, 2020 •

edited

Loading

Hoikas commented Sep 25, 2020

JackBoosY commented Sep 25, 2020

Hoikas commented Sep 25, 2020

JackBoosY commented Sep 27, 2020

Hoikas commented Oct 1, 2020

JackBoosY commented Oct 9, 2020

Hoikas commented Oct 12, 2020

Hoikas commented Nov 3, 2020

JackBoosY commented Nov 4, 2020

[python3] Improve support for embedding and add tools #13556

[python3] Improve support for embedding and add tools #13556

Conversation

Hoikas commented Sep 16, 2020 • edited Loading

PhoebeHui commented Sep 17, 2020

JackBoosY Sep 18, 2020

Choose a reason for hiding this comment

ras0219-msft commented Sep 24, 2020 • edited Loading

Hoikas commented Sep 24, 2020 • edited Loading

Hoikas commented Sep 25, 2020

JackBoosY commented Sep 25, 2020

Hoikas commented Sep 25, 2020

JackBoosY commented Sep 27, 2020

Hoikas commented Oct 1, 2020

JackBoosY commented Oct 9, 2020

Hoikas commented Oct 12, 2020

Hoikas commented Nov 3, 2020

JackBoosY commented Nov 4, 2020

Hoikas commented Sep 16, 2020 •

edited

Loading

ras0219-msft commented Sep 24, 2020 •

edited

Loading

Hoikas commented Sep 24, 2020 •

edited

Loading