Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

android compilation and link flags #71038

Closed
xdegaye mannequin opened this issue Apr 26, 2016 · 19 comments
Closed

android compilation and link flags #71038

xdegaye mannequin opened this issue Apr 26, 2016 · 19 comments
Assignees
Labels
3.7 build The build process and cross-build type-feature A feature request or enhancement

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Apr 26, 2016

BPO 26851
Nosy @Yhg1s, @doko42, @vstinner, @xdegaye, @vadmium, @moreati, @yan12125
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
  • Files
  • build-flags.patch
  • build-flags_2.patch
  • build-flags_3.patch
  • build-flags_3.patch: uploading twice the same file
  • build-flags_4.patch
  • build-flags_5.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/xdegaye'
    closed_at = <Date 2017-01-04.20:58:21.080>
    created_at = <Date 2016-04-26.12:20:11.511>
    labels = ['type-feature', '3.7', 'build']
    title = 'android compilation and link flags'
    updated_at = <Date 2017-03-31.16:36:19.595>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:19.595>
    actor = 'dstufft'
    assignee = 'xdegaye'
    closed = True
    closed_date = <Date 2017-01-04.20:58:21.080>
    closer = 'xdegaye'
    components = ['Cross-Build']
    creation = <Date 2016-04-26.12:20:11.511>
    creator = 'xdegaye'
    dependencies = []
    files = ['42601', '43868', '43875', '43878', '43893', '45101']
    hgrepos = []
    issue_num = 26851
    keywords = ['patch']
    message_count = 19.0
    messages = ['264266', '271169', '271193', '271246', '271275', '271282', '271292', '271353', '271518', '271530', '271543', '271552', '278708', '278712', '284667', '284668', '284739', '284745', '284752']
    nosy_count = 8.0
    nosy_names = ['twouters', 'doko', 'vstinner', 'xdegaye', 'python-dev', 'martin.panter', 'Alex.Willmer', 'yan12125']
    pr_nums = ['552']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue26851'
    versions = ['Python 3.6', 'Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Apr 26, 2016

    The attached patch:

    • Sets the recommended android compilation flags, see: http://developer.android.com/ndk/guides/standalone_toolchain.html#abi. Note that the android toolchains already set the -fpic flag, as shown with:

      arm-linux-androideabi-gcc -v -S main.c 2>&1 | grep main.c

    • Sets the Position independent executables (PIE) flag which is mandatory starting at API level 21 and supported starting with API level 16.

    @xdegaye xdegaye mannequin added build The build process and cross-build type-feature A feature request or enhancement labels Apr 26, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 24, 2016

    The previous patch was using awkwardly, a 'host_cpu' configure argument to specify the arm ABI and missed setting LDFLAGS for the armv7 ABI.
    This patch instead uses the __ARM_ARCH macro defined by each compiler of the Android toolchains:

        echo | ./clang -target armv7-none-linux-androideabi -dM -E - | grep ARM_ARCH
          #define __ARM_ARCH 7
          #define __ARM_ARCH_7A__ 1
        echo | ./clang -target armv5te-none-linux-androideabi -dM -E - | grep ARM_ARCH
          #define __ARM_ARCH 5
          #define __ARM_ARCH_5TE__ 1
        echo | ./arm-linux-androideabi-gcc -march=armv7-a -dM -E - | grep ARM_ARCH
          #define __ARM_ARCH 7
          #define __ARM_ARCH_7A__ 1
        echo | ./arm-linux-androideabi-gcc -dM -E - | grep ARM_ARCH
          #define __ARM_ARCH_5TE__ 1
          #define __ARM_ARCH 5

    Python built with clang for the armv5te target crashes as reported in bpo-27606.

    @xdegaye xdegaye mannequin self-assigned this Jul 24, 2016
    @vadmium
    Copy link
    Member

    vadmium commented Jul 25, 2016

    I left some suggestions and questions on the code review.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 25, 2016

    Thanks for the review and the suggestions Martin :)
    New patch.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 25, 2016

    The first 'build-flags_3.patch' file that was uploaded did not get a review button because ssh://hg@hg.python.org/cpython was down at that time.

    The second upload has been successfull !

    @vstinner
    Copy link
    Member

    vstinner commented Jul 25, 2016

    Note: build-flags_3.patch includes configure.ac and configure. Usually, we suggest to not included generated files (configure) in patches, but push them when the patch is ready.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 25, 2016

    Thanks for the suggestion, I won't include them anymore in patches then.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 26, 2016

    New patch. The sed commands used to evaluate ANDROID_API_LEVEL and _arm_arch do not need to discard comment lines and empty lines in the output of the preprocessor.

    @vadmium
    Copy link
    Member

    vadmium commented Jul 28, 2016

    All the bits that I understand look okay now. :)

    I am still curious what configures the preprocessor to set __ARM_ARCH to 7 (I guess the clang -target argument?), and why we can’t set LDFLAGS at the same time or place. Is it just more convenient this way?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    I am still curious what configures the preprocessor to set __ARM_ARCH to 7 (I guess the clang -target argument?)
    Yes, the -target clang argument or the -march gcc argument.

    and why we can’t set LDFLAGS at the same time or place. Is it just more convenient this way?
    I don't understand the question. LDFLAGS is set at the time we know that __ARM_ARCH is 7, just after the preprocessing is done. Would you set it elsewhere ?
    LDFLAGS is set in configure.ac with always this same idiom: LDFLAGS="$LDFLAGS new_options..." and this setting is done in what seems to be random places in configure.ac.
    CCSHARED is set in one case statement.

    @vadmium
    Copy link
    Member

    vadmium commented Jul 28, 2016

    Where is the code that sets the clang -target argument, or gcc -march? Is it hidden away somewhere in the autoconf code? Do you manually set it with ‘./configure CFLAGS="-target . . ." ’?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jul 28, 2016

    Yes, the packager must use appropriately either CFLAGS CPPFLAGS [1] and LDFLAGS, or CC. I am using:
    CC="clang --sysroot=$(SYSROOT) -target $(TARGET) -gcc-toolchain $(GCC_TOOLCHAIN)".

    [1] See bpo-27453, for the remaining problem upon using CPPFLAGS.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Oct 15, 2016

    See also the clang cross compilation documentation at http://clang.llvm.org/docs/CrossCompilation.html.

    @xdegaye xdegaye mannequin added the 3.7 label Oct 15, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Oct 15, 2016

    This new patch fixes the following:

    • Use BASECFLAGS instead of CCSHARED.
    • '-march=armv7-a' is not set in BASECFLAGS anymore as this is redundant: this option is already set in the configure command line or is implicitly set by the the first component of the triple of the '-target' clang option.
    • Do not set '-mthumb' as this is optional and may be set for the armv7 arch as well. Also this option crashes Python on armv5te when built with clang, bpo-27606.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jan 4, 2017

    New changeset fa2bc63e64c6 by Xavier de Gaye in branch '3.6':
    Issue bpo-26851: Set Android compilation and link flags.
    https://hg.python.org/cpython/rev/fa2bc63e64c6

    New changeset af363b5200ff by Xavier de Gaye in branch 'default':
    Issue bpo-26851: Merge 3.6.
    https://hg.python.org/cpython/rev/af363b5200ff

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jan 4, 2017

    Latest patch committed with the following simplification: BASECFLAGS and LDFLAGS are now set for armv7 at the same place, as suggested by Martin in msg271518.

    @xdegaye xdegaye mannequin closed this as completed Jan 4, 2017
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jan 5, 2017

    In the code review Victor asked the following question:

    I'm a little bit surprised that you need to pass so much options which are specific to the platform. GCC doesn't have a generic option "hello, please compile for my architecture?"

    ARM defines two incompatible floating point ABIS that specify the functions calling conventions, whether to use integer registers (soft floating point ABI) or the floating point registers (hard floating point ABI). The compiler must be told which one to use and whether the hardware FPU may be used if present when the soft floating point ABI has been chosen. This last case is set with the command line option '-mfloat-abi=softfp'. The following two documents describe this feature in details:
    https://wiki.debian.org/ArmHardFloatPort/VfpComparison
    https://wiki.debian.org/ArmHardFloatPort

    The following command prints the command line options, including implicitly predefined ones of the clang compiler in android-ndk-r13b and shows that the default is '-mfloat-abi=soft' for armv7:

    $ ./clang -x c < /dev/null -dM -E - -target armv7-none-linux-androideabi -###
    Android clang version 3.8.256229  (based on LLVM 3.8.256229)
    Target: armv7-none-linux-android
    Thread model: posix
    InstalledDir: /opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/.
     "/home/opt.symlink/android-ndk-r13b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang" "-cc1" "-triple" "armv7-none-linux-android" "-E" "-disable-free" "-disable-llvm-verifier" "-main-file-name" "-" "-mrelocation-model" "pic" "-pic-level" "1" "-mthread-model" "posix" "-mdisable-fp-elim" "-fmath-errno" "-masm-verbose" "-mconstructor-aliases" "-fuse-init-array" "-target-cpu" "cortex-a8" "-target-feature" "+soft-float-abi" "-target-abi" "aapcs-linux" "-mfloat-abi" "soft" "-target-linker-version" "2.24" "-dwarf-column-info" "-debugger-tuning=gdb" "-resource-dir" "/home/opt.symlink/android-ndk-r13b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib64/clang/3.8.256229" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/home/opt.symlink/android-ndk-r13b/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib64/clang/3.8.256229/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdebug-compilation-dir" "/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin" "-ferror-limit" "19" "-fmessage-length" "100" "-femulated-tls" "-fallow-half-arguments-and-returns" "-fno-signed-char" "-fobjc-runtime=gcc" "-fdiagnostics-show-option" "-fcolor-diagnostics" "-dM" "-o" "-" "-x" "c" "-"

    Note that the system libffi must also be compiled with these same flags. The bundled libffi is deprecated in 3.6 by bpo-27976 and has been removed in 3.7 by bpo-27979.

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Jan 5, 2017

    Note that the system libffi must also be compiled with these same flags

    Just tried. With my packaging scripts, CPython on ARM is compiled with -mfloat-abi=softfp -mfpu=vfpv3-d16 while libffi not. test_ctypes pass as usual. Maybe ctypes test suite is not complete?

    And by the way, if all packages on a system should be compiled with the same set of flags, shouldn't those flags specified in build/package scripts rather than Makefile of individual packages?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jan 5, 2017

    Just tried. With my packaging scripts, CPython on ARM is compiled with -mfloat-abi=softfp -mfpu=vfpv3-d16 while libffi not. test_ctypes pass as usual.

    This works as expected, '-mfloat-abi=softfp' and the default '-mfloat-abi=soft' use the same ABI [1]: "Function calls are generated to pass FP arguments (float, double) in integer registers (one for float, a pair of registers for double)".

    [1] https://wiki.debian.org/ArmHardFloatPort/VfpComparison

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 build The build process and cross-build type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants