[Python3] cross build support #27830

rinechran · 2022-11-15T12:49:59Z

Describe the pull request

What does your PR fix?

Python3 cross support
Which triplets are supported/not supported? Have you updated the CI baseline?

all
Does your PR follow the maintainer guide?

Your answer
If you have added/updated a port: Have you run ./vcpkg x-add-version --all and committed the result?

Yes

github-actions

You have modified or added at least one portfile where deprecated functions are used.

If you feel able to do so, please consider migrating them to the new functions:
vcpkg_install_cmake -> vcpkg_cmake_install (from port vcpkg-cmake)
vcpkg_build_cmake -> vcpkg_cmake_build (from port vcpkg-cmake)
vcpkg_configure_cmake -> vcpkg_cmake_configure (Please remove the option PREFER_NINJA) (from port vcpkg-cmake)
vcpkg_fixup_cmake_targets -> vcpkg_cmake_config_fixup (from port vcpkg-cmake-config)
vcpkg_extract_source_archive_ex -> vcpkg_extract_source_archive
vcpkg_build_msbuild -> vcpkg_install_msbuild
vcpkg_copy_tool_dependencies -> vcpkg_copy_tools
vcpkg_apply_patches should be replaced by the PATCHES arguments to the "extract" helpers (e.g. vcpkg_from_github())

In the ports that use the new function, you have to add the corresponding dependencies:

{
  "name": "vcpkg-cmake",
  "host": true
},
{
  "name": "vcpkg-cmake-config",
  "host": true
}

The following files are affected:

ports/python3/portfile.cmake

Osyotr · 2022-11-15T15:15:54Z

Maybe you need to set VCPKG_MAKE_BUILD_TRIPLET in your custom triplet.
https://github.com/Microsoft/vcpkg/blob/master/docs/maintainers/vcpkg_configure_make.md#build_triplet

rinechran · 2022-11-15T15:28:37Z

Are you saying that I should let you do VCPKG_MAKE_BUILD_TRIPLET when it's VCPKG_CROSS COMPILING?

dg0yt · 2022-11-15T18:34:19Z

The first step is to add DETERMINE_BUILD_TRIPLET to vcpkg_cmake_configure. --host is an argument which takes a value, and the proposed keyword will handle this for common setups.
VCPKG_MAKE_BUILD_TRIPLET is a (vcpkg) triplet variable which can override the automatically determined value for less common (or unsupported) setups.

Note that there can be more blockers for cross builds, in particular if the package tries to build some internal tools for the build maching.

Osyotr · 2022-11-15T20:37:17Z

@dg0yt would you kindly help me understand why does BUILD_TRIPLET argument exists if its value depends on toolchain and why DETERMINE_BUILD_TRIPLET is not default?

dg0yt · 2022-11-15T21:14:56Z

@dg0yt would you kindly help me understand why does BUILD_TRIPLET argument exists if its value depends on toolchain and why DETERMINE_BUILD_TRIPLET is not default?

I could only speculate: The authors didn't want to rely on a particular automatic detection, or they didn't want to change the default behaviour when the keyword was added.
Note that AUTOCONFIG also implies DETERMINE_BUILD_TRIPLET.
And there are more quirks. There is no CI coverage for these kinds of cross builds.

rinechran · 2022-11-16T01:03:56Z

@dg0yt would you kindly help me understand why does BUILD_TRIPLET argument exists if its value depends on toolchain and why DETERMINE_BUILD_TRIPLET is not default?

I could only speculate: The authors didn't want to rely on a particular automatic detection, or they didn't want to change the default behaviour when the keyword was added. Note that AUTOCONFIG also implies DETERMINE_BUILD_TRIPLET. And there are more quirks. There is no CI coverage for these kinds of cross builds.

i respect your opinion .the, it seem right to add DETERMINE_BUILD_TRIPLET

Cheney-W · 2022-11-16T06:28:07Z

Please modify the value of "port-version" from 4 to 5 in vcpkg.json and run .\vcpkg x-add-version python3 --overwrite-version

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

PRs must add only one version and must not modify any published versions

When making any changes to a library, the version or port-version in vcpkg.json or CONTROL must be modified.

error: checked-in files for python3 have changed but the version was not updated
version: 3.10.7
old SHA: 047f7eca7d1000287f2389302d5c292251d023f2
new SHA: 7cd6ebf0b3b8ac248e63ae7ccd7f4f8288e924b0
Did you remember to update the version or port version?
Use --overwrite-version to bypass this check
***No files were updated***

You have modified or added at least one portfile where deprecated functions are used.

If you feel able to do so, please consider migrating them to the new functions:
vcpkg_install_cmake -> vcpkg_cmake_install (from port vcpkg-cmake)
vcpkg_build_cmake -> vcpkg_cmake_build (from port vcpkg-cmake)
vcpkg_configure_cmake -> vcpkg_cmake_configure (Please remove the option PREFER_NINJA) (from port vcpkg-cmake)
vcpkg_fixup_cmake_targets -> vcpkg_cmake_config_fixup (from port vcpkg-cmake-config)
vcpkg_extract_source_archive_ex -> vcpkg_extract_source_archive
vcpkg_build_msbuild -> vcpkg_install_msbuild
vcpkg_copy_tool_dependencies -> vcpkg_copy_tools
vcpkg_apply_patches should be replaced by the PATCHES arguments to the "extract" helpers (e.g. vcpkg_from_github())

In the ports that use the new function, you have to add the corresponding dependencies:

{
  "name": "vcpkg-cmake",
  "host": true
},
{
  "name": "vcpkg-cmake-config",
  "host": true
}

The following files are affected:

ports/python3/portfile.cmake

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

PRs must add only one version and must not modify any published versions

When making any changes to a library, the version or port-version in vcpkg.json or CONTROL must be modified.

error: checked-in files for python3 have changed but the version was not updated
version: 3.10.7
old SHA: 7cd6ebf0b3b8ac248e63ae7ccd7f4f8288e924b0
new SHA: 2dccc082ae050da13b2db906c619b14846babef1
Did you remember to update the version or port version?
Use --overwrite-version to bypass this check
***No files were updated***

After committing all other changes, the version database must be updated

git add -u && git commit
git checkout 34d2cf7e62d781f4bcb9c7f44f4d2389f568e92b -- versions
./vcpkg x-add-version --all

Diff

diff --git a/versions/baseline.json b/versions/baseline.json
index a07605c..f811c1b 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -6018,7 +6018,7 @@
     },
     "python3": {
       "baseline": "3.10.7",
-      "port-version": 1
+      "port-version": 0
     },
     "qca": {
       "baseline": "2.3.4",

You have modified or added at least one portfile where deprecated functions are used.

If you feel able to do so, please consider migrating them to the new functions:
vcpkg_install_cmake -> vcpkg_cmake_install (from port vcpkg-cmake)
vcpkg_build_cmake -> vcpkg_cmake_build (from port vcpkg-cmake)
vcpkg_configure_cmake -> vcpkg_cmake_configure (Please remove the option PREFER_NINJA) (from port vcpkg-cmake)
vcpkg_fixup_cmake_targets -> vcpkg_cmake_config_fixup (from port vcpkg-cmake-config)
vcpkg_extract_source_archive_ex -> vcpkg_extract_source_archive
vcpkg_build_msbuild -> vcpkg_install_msbuild
vcpkg_copy_tool_dependencies -> vcpkg_copy_tools
vcpkg_apply_patches should be replaced by the PATCHES arguments to the "extract" helpers (e.g. vcpkg_from_github())

In the ports that use the new function, you have to add the corresponding dependencies:

{
  "name": "vcpkg-cmake",
  "host": true
},
{
  "name": "vcpkg-cmake-config",
  "host": true
}

The following files are affected:

ports/python3/portfile.cmake

github-actions

You have modified or added at least one portfile where deprecated functions are used.

If you feel able to do so, please consider migrating them to the new functions:
vcpkg_install_cmake -> vcpkg_cmake_install (from port vcpkg-cmake)
vcpkg_build_cmake -> vcpkg_cmake_build (from port vcpkg-cmake)
vcpkg_configure_cmake -> vcpkg_cmake_configure (Please remove the option PREFER_NINJA) (from port vcpkg-cmake)
vcpkg_fixup_cmake_targets -> vcpkg_cmake_config_fixup (from port vcpkg-cmake-config)
vcpkg_extract_source_archive_ex -> vcpkg_extract_source_archive
vcpkg_build_msbuild -> vcpkg_install_msbuild
vcpkg_copy_tool_dependencies -> vcpkg_copy_tools
vcpkg_apply_patches should be replaced by the PATCHES arguments to the "extract" helpers (e.g. vcpkg_from_github())

In the ports that use the new function, you have to add the corresponding dependencies:

{
  "name": "vcpkg-cmake",
  "host": true
},
{
  "name": "vcpkg-cmake-config",
  "host": true
}

The following files are affected:

ports/python3/portfile.cmake

github-actions

This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!

PRs must add only one version and must not modify any published versions

When making any changes to a library, the version or port-version in vcpkg.json or CONTROL must be modified.

warning: checked-in files for python3 are unchanged from version 3.10.7#1
-- SHA: 2dccc082ae050da13b2db906c619b14846babef1
-- Did you remember to commit your changes?
***No files were updated***

After committing all other changes, the version database must be updated

git add -u && git commit
git checkout 34d2cf7e62d781f4bcb9c7f44f4d2389f568e92b -- versions
./vcpkg x-add-version --all

Diff

diff --git a/versions/baseline.json b/versions/baseline.json
index a07605c..f811c1b 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -6018,7 +6018,7 @@
     },
     "python3": {
       "baseline": "3.10.7",
-      "port-version": 1
+      "port-version": 0
     },
     "qca": {
       "baseline": "2.3.4",

You have modified or added at least one portfile where deprecated functions are used.

If you feel able to do so, please consider migrating them to the new functions:
vcpkg_install_cmake -> vcpkg_cmake_install (from port vcpkg-cmake)
vcpkg_build_cmake -> vcpkg_cmake_build (from port vcpkg-cmake)
vcpkg_configure_cmake -> vcpkg_cmake_configure (Please remove the option PREFER_NINJA) (from port vcpkg-cmake)
vcpkg_fixup_cmake_targets -> vcpkg_cmake_config_fixup (from port vcpkg-cmake-config)
vcpkg_extract_source_archive_ex -> vcpkg_extract_source_archive
vcpkg_build_msbuild -> vcpkg_install_msbuild
vcpkg_copy_tool_dependencies -> vcpkg_copy_tools
vcpkg_apply_patches should be replaced by the PATCHES arguments to the "extract" helpers (e.g. vcpkg_from_github())

In the ports that use the new function, you have to add the corresponding dependencies:

{
  "name": "vcpkg-cmake",
  "host": true
},
{
  "name": "vcpkg-cmake-config",
  "host": true
}

The following files are affected:

ports/python3/portfile.cmake

rinechran · 2022-11-22T17:27:47Z

The "cpkg x-add-version" version does not go up because of the merge. What should I do?

github-actions

You have modified or added at least one portfile where deprecated functions are used.

If you feel able to do so, please consider migrating them to the new functions:
vcpkg_install_cmake -> vcpkg_cmake_install (from port vcpkg-cmake)
vcpkg_build_cmake -> vcpkg_cmake_build (from port vcpkg-cmake)
vcpkg_configure_cmake -> vcpkg_cmake_configure (Please remove the option PREFER_NINJA) (from port vcpkg-cmake)
vcpkg_fixup_cmake_targets -> vcpkg_cmake_config_fixup (from port vcpkg-cmake-config)
vcpkg_extract_source_archive_ex -> vcpkg_extract_source_archive
vcpkg_build_msbuild -> vcpkg_install_msbuild
vcpkg_copy_tool_dependencies -> vcpkg_copy_tools
vcpkg_apply_patches should be replaced by the PATCHES arguments to the "extract" helpers (e.g. vcpkg_from_github())

In the ports that use the new function, you have to add the corresponding dependencies:

{
  "name": "vcpkg-cmake",
  "host": true
},
{
  "name": "vcpkg-cmake-config",
  "host": true
}

The following files are affected:

ports/python3/portfile.cmake

Cheney-W · 2022-11-23T02:16:54Z

python3 has been update to 3.10.7, please do as below:

Add "port-version": 1, after "version": "3.10.7", in python3/vcpkg.json manually.
git add .
git commit -m "xxxxxxx"
.\vcpkg x-add-version python3 --overwrite-version
git add .
git commit --amend --no-edit

Cheney-W · 2022-11-25T01:32:14Z

Waiting #27973 merged.

rinechran · 2022-12-05T08:34:48Z

Can I close this pr?

Cheney-W · 2022-12-05T10:12:43Z

Can I close this pr?

Sorry, I forgot it.
No. The dependence PR has been merged, I will rerun this PR, if the CI test pass, we will consider merge this PR.

Cheney-W · 2022-12-05T10:12:51Z

/azp run

azure-pipelines · 2022-12-05T10:13:00Z

Azure Pipelines successfully started running 1 pipeline(s).

rinechran · 2022-12-08T12:40:17Z

please check the pr @Cheney-W

Cheney-W · 2022-12-09T02:07:27Z

please check the pr @Cheney-W

This PR is just waiting to be merged.

autoantwort · 2022-12-20T14:25:06Z

This change breaks the build of python:arm64-osx on an arm64 system

rinechran · 2022-12-20T14:40:07Z

It's a flag that only works when crossbuilding, so I don't think it'll affect when building locally on "python:arm64-osx". @autoantwort

autoantwort · 2022-12-20T14:42:31Z

But it does. Now building python fails with

configure: WARNING: unrecognized options: --disable-silent-rules, --enable-static
configure: error: Cross compiling required --host=HOST-TUPLE and --build=ARCH

And before this change even cross building already works, now it also fails with the same error.

rinechran · 2022-12-20T14:44:26Z

Ha...
I think this error may be caused by the DETERMINE_BUILD_TRIPLET flag

autoantwort · 2022-12-20T14:49:15Z

Do you have an example of a build failure that was fixed by this PR or do we only have examples of things that are now broken after this PR?

rinechran · 2022-12-20T14:50:21Z

I'm not sure about that.

autoantwort · 2022-12-20T14:52:20Z

So what was the motivation behind creating this PR?

rinechran · 2022-12-20T14:56:54Z

An error occurs when cross-build python.
I added a version of the library by referring to the site below
https://bugs.python.org/issue3754 , https://unix.stackexchange.com/questions/342925/cross-compiling-python

autoantwort · 2022-12-20T15:02:39Z

Can you give an example from which to which triplet you wanted to cross compile where it failed?

Osyotr · 2022-12-20T15:35:45Z

DETERMINE_BUILD_TRIPLET shall be eradicated (maybe #28374 could help).
ac_cv_file__dev_ptmx and ac_cv_file__dev_ptc values should come from your custom triplet.

autoantwort · 2022-12-20T15:44:05Z

Even with #28374 I get the following error when I try to build python3:arm64-osx on my arm64-osx mac:

configure: WARNING: unrecognized options: --disable-silent-rules, --enable-static
configure: error: cross build not supported for aarch64-apple-darwin

dpogue · 2022-12-22T06:36:22Z

@autoantwort Ironically, I opened that PR specifically to try to work around issues cross-compiling arm64-osx Python from an x86_64 macOS builder, but alas (as you know) Python doesn't support cross-compiling on macOS: python/cpython#90905 😞

rinechran added 2 commits November 15, 2022 21:46

[python3] Python3 cross support

d346212

[python3] update python3 port versions

8f5dde3

rinechran changed the title ~~Python3 cross support~~ [Python3] cross build support Nov 15, 2022

github-actions bot previously approved these changes Nov 15, 2022

View reviewed changes

FrankXie05 assigned Cheney-W Nov 16, 2022

Cheney-W added category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist requires:author-response labels Nov 16, 2022

Merge branch 'master' into python3

13e5c28

rinechran dismissed github-actions[bot]’s stale review via 13e5c28 November 22, 2022 15:32

github-actions bot requested changes Nov 22, 2022

View reviewed changes

[python3] : add option when cross build

27ac19b

github-actions bot requested changes Nov 22, 2022

View reviewed changes

[python3] port-verionsing

04789f6

github-actions bot previously approved these changes Nov 22, 2022

View reviewed changes

python3 : port-version to change 1

1e42727

rinechran dismissed github-actions[bot]’s stale review via 1e42727 November 22, 2022 15:56

github-actions bot requested changes Nov 22, 2022

View reviewed changes

[python3] versioning

393dc27

github-actions bot previously approved these changes Nov 22, 2022

View reviewed changes

[python3] change baseline update

cecb206

rinechran dismissed github-actions[bot]’s stale review via cecb206 November 23, 2022 11:38

Cheney-W added depends:different-pr This PR or Issue depends on a PR which has been filed and removed requires:author-response labels Nov 25, 2022

Cheney-W added info:reviewed Pull Request changes follow basic guidelines and removed depends:different-pr This PR or Issue depends on a PR which has been filed labels Dec 5, 2022

dan-shaw merged commit 1937b15 into microsoft:master Dec 15, 2022

Hoikas mentioned this pull request Dec 16, 2022

Run macOS builds with vcpkg again H-uru/Plasma#1276

Merged

rinechran deleted the python3 branch December 19, 2022 10:31

autoantwort mentioned this pull request Dec 20, 2022

[Python3] Revert "Python3 cross build support" #28468

Merged

This was referenced May 15, 2023

[python3] Build error (arm64-android) #30501

Closed

[python3] Enable cross-compiling for non-macOS *nix platforms #31497

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python3] cross build support #27830

[Python3] cross build support #27830

rinechran commented Nov 15, 2022

github-actions bot left a comment

Osyotr commented Nov 15, 2022

rinechran commented Nov 15, 2022 •

edited

Loading

dg0yt commented Nov 15, 2022

Osyotr commented Nov 15, 2022

dg0yt commented Nov 15, 2022

rinechran commented Nov 16, 2022

Cheney-W commented Nov 16, 2022

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

rinechran commented Nov 22, 2022

github-actions bot left a comment

Cheney-W commented Nov 23, 2022

Cheney-W commented Nov 25, 2022

rinechran commented Dec 5, 2022

Cheney-W commented Dec 5, 2022

Cheney-W commented Dec 5, 2022

azure-pipelines bot commented Dec 5, 2022

rinechran commented Dec 8, 2022

Cheney-W commented Dec 9, 2022

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022 •

edited

Loading

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022

autoantwort commented Dec 20, 2022

Osyotr commented Dec 20, 2022

autoantwort commented Dec 20, 2022

dpogue commented Dec 22, 2022

[Python3] cross build support #27830

[Python3] cross build support #27830

Conversation

rinechran commented Nov 15, 2022

What does your PR fix?

Which triplets are supported/not supported? Have you updated the CI baseline?

Does your PR follow the maintainer guide?

If you have added/updated a port: Have you run ./vcpkg x-add-version --all and committed the result?

github-actions bot left a comment

Choose a reason for hiding this comment

Osyotr commented Nov 15, 2022

rinechran commented Nov 15, 2022 • edited Loading

dg0yt commented Nov 15, 2022

Osyotr commented Nov 15, 2022

dg0yt commented Nov 15, 2022

rinechran commented Nov 16, 2022

Cheney-W commented Nov 16, 2022

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

rinechran commented Nov 22, 2022

github-actions bot left a comment

Choose a reason for hiding this comment

Cheney-W commented Nov 23, 2022

Cheney-W commented Nov 25, 2022

rinechran commented Dec 5, 2022

Cheney-W commented Dec 5, 2022

Cheney-W commented Dec 5, 2022

azure-pipelines bot commented Dec 5, 2022

rinechran commented Dec 8, 2022

Cheney-W commented Dec 9, 2022

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022 • edited Loading

autoantwort commented Dec 20, 2022

rinechran commented Dec 20, 2022

autoantwort commented Dec 20, 2022

Osyotr commented Dec 20, 2022

autoantwort commented Dec 20, 2022

dpogue commented Dec 22, 2022

If you have added/updated a port: Have you run `./vcpkg x-add-version --all` and committed the result?

rinechran commented Nov 15, 2022 •

edited

Loading

rinechran commented Dec 20, 2022 •

edited

Loading