Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cross compilation of third-party extension modules #73019

Open
xdegaye mannequin opened this issue Nov 29, 2016 · 12 comments
Open

cross compilation of third-party extension modules #73019

xdegaye mannequin opened this issue Nov 29, 2016 · 12 comments
Labels
3.7 build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Nov 29, 2016

BPO 28833
Nosy @warsaw, @doko42, @vstinner, @merwok, @pmp-p, @vadmium, @zware, @dstufft, @moreati, @yan12125
PRs
  • bpo-28833: Fix cross-compilation of third-party extension modules #17420
  • Files
  • cross-build-extension.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2016-11-29.14:51:40.336>
    labels = ['type-bug', '3.7', 'build']
    title = 'cross compilation of third-party extension modules'
    updated_at = <Date 2019-12-10.08:01:23.697>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-12-10.08:01:23.697>
    actor = 'xdegaye'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Cross-Build']
    creation = <Date 2016-11-29.14:51:40.336>
    creator = 'xdegaye'
    dependencies = []
    files = ['45688']
    hgrepos = []
    issue_num = 28833
    keywords = ['patch']
    message_count = 12.0
    messages = ['281993', '281996', '281998', '282044', '282103', '282104', '282141', '282144', '282147', '282155', '282157', '357663']
    nosy_count = 10.0
    nosy_names = ['barry', 'doko', 'vstinner', 'eric.araujo', 'pmpp', 'martin.panter', 'zach.ware', 'dstufft', 'Alex.Willmer', 'yan12125']
    pr_nums = ['17420']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue28833'
    versions = ['Python 3.6', 'Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 29, 2016

    With this patch, cross compiling a third-party extension module is done with the command:

    XBUILD_PYTHON_DIR=/path/to/python/dir python setup.py build

    where XBUILD_PYTHON_DIR is the location of the directory of the cross-compiled python executable.
    It may be:
    a) The build tree, which is the source tree when the cross compilation is not out of the source tree.
    b) '$DESTDIR/$exec_prefix/bin' when the cross built python has been installed with 'make DESTDIR=/some/path install'. In that case 'prefix' and 'exec_prefix' may be different.
    c) When the result of the cross compilation has been manually copied (for example to /some/path) and if 'prefix' and 'exec_prefix' are identical, this is /some/path/bin.
    In case b), one can use the 'install' setup.py command instead of the 'build' command, to build and install the extension module to '$DESTDIR/$exec_prefix/lib/python$VERSION/site-packages' with the appropriate 'egg-info' file.

    The patch uses the 'python-config' shell script (created at bpo-16235 [1]) that is initialized with the proper variables at the configure stage to provide the minimum information required by the sysconfig module to create (with the option --generate-posix-vars) the sysconfigdata file or to import the sysconfigdata module.
    The patch also fixes bpo-22724 [2] as sys.path is only modified during the time needed to import the sysconfigdata module.

    The patch fixes two minor problems:

    • '_PYTHON_HOST_PLATFORM' in Makefile.pre.in was not added to the sysconfig variables as intended, since those variables may not be prefixed with an underscore.
    • The sysconfigdata file name was terminated with a dangling underscore when 'multiarch' is not defined.
      Patch also tested with pyephem on an Android emulator.
      The patch misses the documentation.
      Please run autoconf after installing the patch.

    [1] bpo-16235: add python-config.sh for use during cross compilation
    http://bugs.python.org/issue16235
    [2] bpo-22724: byte-compile fails for cross-builds
    http://bugs.python.org/issue22724

    @xdegaye xdegaye mannequin self-assigned this Nov 29, 2016
    @xdegaye xdegaye mannequin added 3.7 build The build process and cross-build type-bug An unexpected behavior, bug, or error labels Nov 29, 2016
    @doko42
    Copy link
    Member

    doko42 commented Nov 29, 2016

    This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.

    Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).

    I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).

    The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.

    Minor issue: please s/xbuild/cross_build/.

    @doko42
    Copy link
    Member

    doko42 commented Nov 29, 2016

    • The sysconfigdata file name was terminated with a dangling
      underscore when 'multiarch' is not defined.

    That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 29, 2016

    This approach will not work with a "multiarch" enabled environment, and break cross builds on Debian and Ubuntu.

    No, the patch does not break cross builds on Debian and Ubuntu, unless you can demonstrate it does.

    Afaics, the proposal assumes that the python executable for the target architecture is installed (which it is not for the multiarch cross-build layout), and that each target architecture has to be identified by it's own directory tree (again, not in the multiarch environment, you can install multiple targets into the same path).

    No, you are mistaken, the path name of the build tree may be used to cross build third-party extension modules as stated in case a) of msg281993, and this should also work with debian. BTW the same code is run to cross build a standard library extension module and a third-party extension module, and obviously python is not yet installed by the time the standard library extension modules are built ;-)

    The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.

    It is not clear why, because debian has this multiarch design, this should constrain our build system to follow their paradigm. After all, debian has already found some ways to cross-build the extension modules for their supported multiarch platforms since it is not possible, as of today, with upstream CPython.

    Minor issue: please s/xbuild/cross_build/.

    Agreed.

    > * The sysconfigdata file name was terminated with a dangling
    > underscore when 'multiarch' is not defined.

    That only solves part of the problem in that the kernel/os version gets encoded as well, e.g. gnukfreebsd9, gnukfreebsd10, which is nasty when the version of your runtime and build time kernel differ.

    No idea what is the problem you are refering to. It cannot be the "dangling underscore" problem since this is a cosmetic issue. Please explain.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 30, 2016

    I don't think that identifying the target by some path is the right way to go, and you should be able to identify the target by giving the target triplet as used by the configure parameters and then deduce the location from the target (or have an explicit location given).

    The idea here is to use the platform identifier which we already had in the trunk before the PLATDIR was removed (the multiarch id or if not defined the platform). So by having a <target> specifier, you could deduce a path, or a <target>-python-config executable and you're done. No need to know about a path directly. Of course python cross builds would have to install the <target>-python-config executable or symlink.

    Hum, you still need to provide the native python interpreter with the _path_ to the <target>-python-config executable that can be anywhere on the file system (its location is relative to the cross compiled build, not the native interpreter), so it seems that this contradicts your previous sentence saying "I don't think that identifying the target by some path is the right way to go".

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 30, 2016

    So I suggest we start with this patch as it works for:

    • The standard library extension modules (for debian as well).
    • The third-party extension modules on platforms that have multiarch defined and on platforms that do not have multiarch defined (on debian systems one can only use the build directory for XBUILD_PYTHON_DIR, the other systems can use the three options described in msg281993).

    Then we can later extend the semantics of XBUILD_PYTHON_DIR to include the multiarch triplet if this is necessary for debian systems, unless the debian developers want to continue using the system they have developed and are currently using to cross compile third-party extension modules.

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Dec 1, 2016

    Well, cross compiling extension modules already works fine...

    $ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr python3.7 setup.py build_ext
    running build_ext
    building 'xx' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE -fno-integrated-as -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -fPIE -fno-integrated-as -fPIC -I/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/include/python3.7m -c xxmodule.c -o build/temp.linux-x86_64-3.7/xxmodule.o
    creating build/lib.linux-x86_64-3.7
    /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -shared -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie -L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib -target aarch64-none-linux-android -gcc-toolchain /opt/android-ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64 --sysroot=/opt/android-ndk/platforms/android-21/arch-arm64/usr -pie -L/home/yen/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr/lib build/temp.linux-x86_64-3.7/xxmodule.o -o build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so
    clang: warning: argument unused during compilation: '-pie'
    clang: warning: argument unused during compilation: '-pie'

    And it's running fine, too:

    $ adb push build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so /data/local/tmp
    [100%] /data/local/tmp/xx.cpython-37m-aarch64-linux-android.so
    $ adb shell
    shell@ASUS_Z00E_2:/ $ cd /data/local/tmp
    shell@ASUS_Z00E_2:/data/local/tmp $ . ./python3/tools/env.sh
    shell@ASUS_Z00E_2:/data/local/tmp $ python3.7m
    Python 3.7.0a0 (default:3d660ed2a60e+, Nov 23 2016, 20:22:14) 
    [GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import xx
    >>> xx
    <module 'xx' from '/data/local/tmp/xx.cpython-37m-aarch64-linux-android.so'>
    >>>

    Here's my xxmodule.c:

    #include "Python.h"

    PyDoc_STRVAR(module_doc,
    "This is a template module just for instruction.");

    static struct PyModuleDef xxmodule = {
        PyModuleDef_HEAD_INIT,
        "xx",
        module_doc,
        0,
        NULL,
        NULL,
        NULL,
        NULL,
        NULL
    };

    PyMODINIT_FUNC
    PyInit_xx(void)
    {
    return PyModuleDef_Init(&xxmodule);
    }

    And setup.py:

    from distutils.core import setup, Extension
    
    module1 = Extension('xx',
                        sources=['xxmodule.c'])
    
    setup(name='PackageName',
          version='1.0',
          description='This is a demo package',
          ext_modules=[module1])

    Both the host and target Python are built from a6e59a2e880e

    There's a little bug in the built filename: build/lib.linux-x86_64-3.7/xx.cpython-37m-aarch64-linux-android.so, which should be build/lib.linux-aarch64-3.7/xx.cpython-37m-aarch64-linux-android.so.

    Also, I guess there may be quirks if NDK or the cross-built CPython is at different locations than built-time configurations. Those issues can be left to the future.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 1, 2016

    $ _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_aarch64-linux-android PYTHONHOME=~/Projects/python3-android/build/21-aarch64-linux-android-4.9/usr python3.7 setup.py build_ext
    >

    Variables prefixed by an underscore such as '_PYTHON_SYSCONFIGDATA_NAME' are private, please do not use them, they may be changed or removed at any time.

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Dec 1, 2016

    I know. I guess it can be determined without manually specifying but haven't investigated into details. Maybe some patches are necessary.

    @doko42
    Copy link
    Member

    doko42 commented Dec 1, 2016

    again, I don't think relying on a specific target path for a cross target is a good idea. and now deciding that the last possibility to use a target id to identify is better is internal doesn't make it better. I'd appreciate if we could sit together where your work will lead to, and see how we accomplish everybody's goals.

    @doko42
    Copy link
    Member

    doko42 commented Dec 1, 2016

    Hum, you still need to provide the native python interpreter
    with the _path_ to the <target>-python-config executable that
    can be anywhere on the file system

    No, it's found in the same path. No contradiction.

    So I suggest we start with this patch as it works for:

    Please do not.

    Then we can later extend the semantics of XBUILD_PYTHON_DIR

    Again, I think that's the wrong way to go rely on any path as the primary id for the target, then "extending" on it makes it worse.

    @xdegaye xdegaye mannequin removed their assignment Feb 3, 2017
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 30, 2019

    PR 17420 fixes cross-compilation of third-party extension modules.

    The PYTHON_PROJECT_BASE environment variable is the path to the directory where Python has been cross-compiled. It is used by the native python interpreter to find the target sysconfigdata module.

    For example the following command builds a wheel file to be transfered and installed with pip on the target platform, provided the native python interpreter and the cross-compiled one both have the wheel package installed:

      $ PYTHON_PROJECT_BASE=/path/to/builddir python setup.py bdist_wheel

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 build The build process and cross-build type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant