Support PGO in cross-compiles when transparent emulation is available

[zanieb]

A while ago, in 1abe1c5 (from https://bugs.python.org/issue14330) configure was updated to unset RUNSHARED when cross compiling. Unfortunately, this has the effect of disabling PGO during cross-compiles.

In python-build-standalone we patch this out so we can run PGO as we do frequent cross-compilation on systems that support emulated execution of the target binary, e.g., x86-64 macOS can be run on aarch64 via Rosetta.

I'd like to upstream this patch (see astral-sh/python-build-standalone#864) in a generalized form.

This is a subset of #113123 which is about generalized support for PGO in cross-compiles and focuses on setting a variable to wrapper program allowing emulated execution. I intend to look into that separately.

Linked PRs

• gh-141926 : Add --enable-cross-run configure option to avoid clearing RUNSHARED on cross-compile #141927
• gh-141926: Do not unset RUNSHARED when cross-compiling #141958

[zanieb]

(sorry I managed to submit the issue in the middle of typing)

[zanieb]

This may be already supported in some subset of use-cases? e.g., when libpython is statically linked? macOS might also be a bad example, as those aren't always considered cross-builds? I'm going to do some checks and share a basic reproduction.

[zanieb]

Well, the simple reproduction for me is

# Build a host Python
./configure \
    --prefix $(pwd)/build-host \
    --without-ensurepip \
    py_cv_module__hashlib=n/a \
    py_cv_module__ssl=n/a
make -j8
make install

# Attempt to cross-compile an x86-64 Python
./configure \
    --build=aarch64-apple-darwin \
    --host=x86_64-apple-darwin \
    --prefix=$(pwd)/build \
    --enable-shared \
    --enable-optimizations \
    --with-build-python=$(pwd)/build-host/bin/python3 \
    ac_cv_lib_intl_textdomain=no \
    MACHDEP=darwin \
    ac_sys_system=Darwin \
    ac_sys_release=24.6.0 \
    ac_cv_func_ptsname_r=no \
    ac_cv_func_explicit_bzero=no \
    --with-dbmliborder=ndbm \
    cross_compiling=yes \
    ac_cv_buggy_getaddrinfo=no \
    ac_cv_file__dev_ptc=no \
    ac_cv_file__dev_ptmx=no \
    ac_cv_pthread=yes \
    ac_cv_aligned_required=no
make -j8

which fails with

LLVM_PROFILE_FILE="/Users/zb/workspace/cpython/code-%p.profclangr"  ./python.exe -m test --pgo --timeout=
dyld[5686]: Library not loaded: /Users/zb/workspace/cpython/install/lib/libpython3.15.dylib
  Referenced from: <51368F0A-5014-3695-8ACB-CDBC35555AFB> /Users/zb/workspace/cpython/python.exe
  Reason: tried: '/Users/zb/workspace/cpython/install/lib/libpython3.15.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/zb/workspace/cpython/install/lib/libpython3.15.dylib' (no such file), '/Users/zb/workspace/cpython/install/lib/libpython3.15.dylib' (no such file)
make: *** [profile-run-stamp] Abort trap: 6

because RUNSHARED is cleared. So on HEAD, PGO is not "disabled", but is broken.

[zanieb]

And, as I suspected, without --enable-shared, PGO works fine — because you don't need RUNSHARED.

[zanieb]

I wonder if we should just stop unsetting RUNSHARED entirely as is done in the python-build-standalone patch? It seems like we'll attempt to run PGO regardless. This would be an alternative to my opt-in approach at #141927 but seems cleaner?

[brettcannon]

I wonder if we should just stop unsetting RUNSHARED entirely as is done in the python-build-standalone patch?

If you're not unsetting it and it's working for PBS and it's passing CI here then it seems like a reasonable thing to do.