-
Notifications
You must be signed in to change notification settings - Fork 729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Load compilation options from _sysconfigdata_*.py file #1095
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❤️ thank you very much for this! It looks like it was simpler than I feared, which is pleasing to see :)
I will probably contribute to https://github.com/davidhewitt/python-config in order to refactor most of the python configuration logic there. It should also improve the debugability of the build script.
Sounds great. What's your strategy for doing this? Maybe let's:
- get a WIP implementation we're happy with in this PR
- port that functionality over to
python-config
in a separate PR - do one final PR to refactor the build script
Its not often you hear these words when it comes to compilation and dynamic linking As for the porting strategy, thats pretty much what I was planing on doing. I'll probably only have time to put into it next weekend. |
4589f84
to
9a41219
Compare
@davidhewitt What is the need for the because the new config map generated by the `_sysconfigdata_` is hugerustc --crate-name pyo3 --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 --cfg 'feature="cross-compile"' --cfg 'feature="ctor"' --cfg 'feature="default"' --cfg 'feature="indoc"' --cfg 'feature="inventory"' --cfg 'feature="macros"' --cfg 'feature="paste"' --cfg 'feature="pyo3cls"' --cfg 'feature="regex"' --cfg 'feature="unindent"' --cfg 'feature="walkdir"' -C metadata=4f5db729a86d41e0 -C extra-filename=-4f5db729a86d41e0 --out-dir /home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps --target arm-linux-androideabi -C incremental=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/incremental -L dependency=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps -L dependency=/home/rene/git/perso/pyo3/target/debug/deps --extern ctor=/home/rene/git/perso/pyo3/target/debug/deps/libctor-d23edd316d8286d6.so --extern indoc=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps/libindoc-4782a3271805adbd.rmeta --extern inventory=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps/libinventory-e88ee21875d02d87.rmeta --extern libc=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps/liblibc-e4daaed138e3377f.rmeta --extern parking_lot=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps/libparking_lot-f8127f7c92732ae6.rmeta --extern paste=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps/libpaste-fde5f3099432a925.rmeta --extern pyo3cls=/home/rene/git/perso/pyo3/target/debug/deps/libpyo3cls-fc433adb7e94fbec.so --extern unindent=/home/rene/git/perso/pyo3/target/arm-linux-androideabi/debug/deps/libunindent-3007c61f26869942.rmeta -L native=/home/rene/git/perso/python3.7_armv7/android-build --cfg Py_3_5 --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3 --cfg 'py_sys_config="CC"' --cfg 'py_sys_config="INSTSONAME"' --cfg 'py_sys_config="RETSIGTYPE"' --cfg 'py_sys_config="LIBSUBDIRS"' --cfg 'py_sys_config="HAVE_DIRFD"' --cfg 'py_sys_config="HAVE_GETPEERNAME"' --cfg 'py_sys_config="HAVE_SIGNAL_H"' --cfg 'py_sys_config="POBJS"' --cfg 'py_sys_config="HAVE_SPAWN_H"' --cfg 'py_sys_config="HAVE_CLOCK_GETTIME"' --cfg 'py_sys_config="HAVE_SETRESGID"' --cfg 'py_sys_config="SCRIPTDIR"' --cfg 'py_sys_config="DLINCLDIR"' --cfg 'py_sys_config="HAVE_ROUND"' --cfg 'py_sys_config="HAVE_TRUNCATE"' --cfg 'py_sys_config="HAVE_SYS_RESOURCE_H"' --cfg 'py_sys_config="HAVE_DECL_RTLD_NOLOAD"' --cfg 'py_sys_config="TESTPYTHONOPTS"' --cfg 'py_sys_config="VERSION"' --cfg 'py_sys_config="HAVE_FCHOWN"' --cfg 'py_sys_config="BASEMODLIBS"' --cfg 'py_sys_config="HAVE_DLOPEN"' --cfg 'py_sys_config="HAVE_UNLINKAT"' --cfg 'py_sys_config="HAVE_FSTATAT"' --cfg 'py_sys_config="CONFIG_ARGS"' --cfg 'py_sys_config="HAVE_MKFIFO"' --cfg 'py_sys_config="INSTALL_DATA"' --cfg 'py_sys_config="HAVE_ACCEPT4"' --cfg 'py_sys_config="HAVE_GETPAGESIZE"' --cfg 'py_sys_config="LIBS"' --cfg 'py_sys_config="UPDATE_FILE"' --cfg 'py_sys_config="INCLDIRSTOMAKE"' --cfg 'py_sys_config="abs_srcdir"' --cfg 'py_sys_config="SIZEOF__BOOL"' --cfg 'py_sys_config="ABIFLAGS"' --cfg 'py_sys_config="HAVE_EPOLL"' --cfg 'py_sys_config="DTRACE_OBJS"' --cfg 'py_sys_config="TESTTIMEOUT"' --cfg 'py_sys_config="HAVE_SETEUID"' --cfg 'py_sys_config="HAVE_UNSETENV"' --cfg 'py_sys_config="HAVE_LGAMMA"' --cfg 'py_sys_config="TIME_WITH_SYS_TIME"' --cfg 'py_sys_config="GITBRANCH"' --cfg 'py_sys_config="BLDSHARED"' --cfg 'py_sys_config="HAVE_FORKPTY"' --cfg 'py_sys_config="READELF"' --cfg 'py_sys_config="HAVE_FSTATVFS"' --cfg 'py_sys_config="HAVE_FCHMODAT"' --cfg 'py_sys_config="CONFIGURE_CPPFLAGS"' --cfg 'py_sys_config="HAVE_FDATASYNC"' --cfg 'py_sys_config="SIZEOF_UINTPTR_T"' --cfg 'py_sys_config="TESTRUNNER"' --cfg 'py_sys_config="HAVE_WMEMCMP"' --cfg 'py_sys_config="HAVE_SYS_SELECT_H"' --cfg 'py_sys_config="HAVE_FUTIMES"' --cfg 'py_sys_config="COREPYTHONPATH"' --cfg 'py_sys_config="HAVE_LIBDL"' --cfg 'py_sys_config="HAVE_MEMRCHR"' --cfg 'py_sys_config="CFLAGS_NODIST"' --cfg 'py_sys_config="HAVE_DECL_ISINF"' --cfg 'py_sys_config="HAVE_STRLCPY"' --cfg 'py_sys_config="HAVE_WAITPID"' --cfg 'py_sys_config="SHELL"' --cfg 'py_sys_config="HAVE_READLINK"' --cfg 'py_sys_config="STRIPFLAG"' --cfg 'py_sys_config="HAVE_LINUX_CAN_H"' --cfg 'py_sys_config="HAVE_SETRESUID"' --cfg 'py_sys_config="HAVE_PIPE2"' --cfg 'py_sys_config="HAVE_ALIGNED_REQUIRED"' --cfg 'py_sys_config="HAVE_SIGINFO_T_SI_BAND"' --cfg 'py_sys_config="HAVE_GETADDRINFO"' --cfg 'py_sys_config="HAVE_SYS_RANDOM_H"' --cfg 'py_sys_config="HAVE_SYS_UIO_H"' --cfg 'py_sys_config="HAVE_SYS_POLL_H"' --cfg 'py_sys_config="COVERAGE_REPORT"' --cfg 'py_sys_config="HAVE_MKFIFOAT"' --cfg 'py_sys_config="SIZEOF_INT"' --cfg 'py_sys_config="VPATH"' --cfg 'py_sys_config="HAVE_WAIT4"' --cfg 'py_sys_config="HAVE_ACOSH"' --cfg 'py_sys_config="HAVE_DECL_RTLD_NOW"' --cfg 'py_sys_config="BUILDEXE"' --cfg 'py_sys_config="HAVE_ERRNO_H"' --cfg 'py_sys_config="HAVE_DECL_RTLD_GLOBAL"' --cfg 'py_sys_config="AR"' --cfg 'py_sys_config="MULTIARCH_CPPFLAGS"' --cfg 'py_sys_config="BUILD_GNU_TYPE"' --cfg 'py_sys_config="TCLTK_INCLUDES"' --cfg 'py_sys_config="HAVE_TERMIOS_H"' --cfg 'py_sys_config="HAVE_CLOCK_SETTIME"' --cfg 'py_sys_config="HAVE_FDOPENDIR"' --cfg 'py_sys_config="HAVE_STRINGS_H"' --cfg 'py_sys_config="CONFIGURE_LDFLAGS"' --cfg 'py_sys_config="HAVE_LANGINFO_H"' --cfg 'py_sys_config="HAVE_MKNODAT"' --cfg 'py_sys_config="HAVE_DECL_RTLD_NODELETE"' --cfg 'py_sys_config="HAVE_X509_VERIFY_PARAM_SET1_HOST"' --cfg 'py_sys_config="HAVE_INTTYPES_H"' --cfg 'py_sys_config="HAVE_PREAD"' --cfg 'py_sys_config="MACHDESTLIB"' --cfg 'py_sys_config="HAVE_ERF"' --cfg 'py_sys_config="HAVE_SYMLINK"' --cfg 'py_sys_config="HAVE_SIGACTION"' --cfg 'py_sys_config="PY3LIBRARY"' --cfg 'py_sys_config="HAVE_PROTOTYPES"' --cfg 'py_sys_config="HAVE_SETPGRP"' --cfg 'py_sys_config="HAVE_FACCESSAT"' --cfg 'py_sys_config="HAVE_GETPGID"' --cfg 'py_sys_config="LLVM_PROF_ERR"' --cfg 'py_sys_config="HAVE_DYNAMIC_LOADING"' --cfg 'py_sys_config="CCSHARED"' --cfg 'py_sys_config="PY_CFLAGS_NODIST"' --cfg 'py_sys_config="HAVE_PTHREAD_GETCPUCLOCKID"' --cfg 'py_sys_config="HAVE_GETRANDOM_SYSCALL"' --cfg 'py_sys_config="HAVE_DEV_PTMX"' --cfg 'py_sys_config="HAVE_LONG_DOUBLE"' --cfg 'py_sys_config="SIZEOF_VOID_P"' --cfg 'py_sys_config="HAVE_TM_ZONE"' --cfg 'py_sys_config="DTRACE_HEADERS"' --cfg 'py_sys_config="HAVE_SYS_FILE_H"' --cfg 'py_sys_config="MACHDEP_OBJS"' --cfg 'py_sys_config="CFLAGS"' --cfg 'py_sys_config="WITH_PYMALLOC"' --cfg 'py_sys_config="HAVE_SIGALTSTACK"' --cfg 'py_sys_config="HAVE_LINUX_CAN_BCM_H"' --cfg 'py_sys_config="HAVE_OPENPTY"' --cfg 'py_sys_config="HAVE_EPOLL_CREATE1"' --cfg 'py_sys_config="OPENSSL_LIBS"' --cfg 'py_sys_config="HAVE_CLOCK_GETRES"' --cfg 'py_sys_config="HAVE_GETRESUID"' --cfg 'py_sys_config="PURIFY"' --cfg 'py_sys_config="HAVE_DLFCN_H"' --cfg 'py_sys_config="MODULE_OBJS"' --cfg 'py_sys_config="ENSUREPIP"' --cfg 'py_sys_config="HAVE_FCHOWNAT"' --cfg 'py_sys_config="HAVE_FCNTL_H"' --cfg 'py_sys_config="HAVE_STRUCT_STAT_ST_BLKSIZE"' --cfg 'py_sys_config="PYTHONFRAMEWORK"' --cfg 'py_sys_config="SIZEOF_OFF_T"' --cfg 'py_sys_config="SIZEOF_TIME_T"' --cfg 'py_sys_config="HAVE_SEM_TIMEDWAIT"' --cfg 'py_sys_config="SHLIBS"' --cfg 'py_sys_config="HAVE_SEM_GETVALUE"' --cfg 'py_sys_config="HAVE_SNPRINTF"' --cfg 'py_sys_config="PY_FORMAT_SIZE_T"' --cfg 'py_sys_config="HAVE_TEMPNAM"' --cfg 'py_sys_config="HAVE_WAITID"' --cfg 'py_sys_config="SIZEOF_PID_T"' --cfg 'py_sys_config="SIZEOF_SHORT"' --cfg 'py_sys_config="RUNSHARED"' --cfg 'py_sys_config="HAVE_BUILTIN_ATOMIC"' --cfg 'py_sys_config="BYTESTR_DEPS"' --cfg 'py_sys_config="CONFINCLUDEDIR"' --cfg 'py_sys_config="HAVE_BROKEN_SEM_GETVALUE"' --cfg 'py_sys_config="HAVE_SYS_STAT_H"' --cfg 'py_sys_config="HAVE_LINKAT"' --cfg 'py_sys_config="DESTLIB"' --cfg 'py_sys_config="HAVE_LUTIMES"' --cfg 'py_sys_config="HAVE_PTHREAD_SIGMASK"' --cfg 'py_sys_config="HAVE_READLINKAT"' --cfg 'py_sys_config="IO_H"' --cfg 'py_sys_config="DESTSHARED"' --cfg 'py_sys_config="HAVE_GETHOSTBYNAME_R_6_ARG"' --cfg 'py_sys_config="HAVE_POSIX_FADVISE"' --cfg 'py_sys_config="HAVE_SIGRELSE"' --cfg 'py_sys_config="HAVE_TIMES"' --cfg 'py_sys_config="BUILDPYTHON"' --cfg 'py_sys_config="SUBDIRS"' --cfg 'py_sys_config="HAVE_POSIX_FALLOCATE"' --cfg 'py_sys_config="HAVE_SYS_XATTR_H"' --cfg 'py_sys_config="HAVE_DEVICE_MACROS"' --cfg 'py_sys_config="MANDIR"' --cfg 'py_sys_config="SYS_SELECT_WITH_SYS_TIME"' --cfg 'py_sys_config="HAVE_STRING_H"' --cfg 'py_sys_config="BINDIR"' --cfg 'py_sys_config="HAVE_LOG2"' --cfg 'py_sys_config="OPENSSL_LDFLAGS"' --cfg 'py_sys_config="HAVE_CHOWN"' --cfg 'py_sys_config="HAVE_GETHOSTBYNAME_R"' --cfg 'py_sys_config="HAVE_SYS_SENDFILE_H"' --cfg 'py_sys_config="WITH_THREAD"' --cfg 'py_sys_config="LLVM_PROF_MERGER"' --cfg 'py_sys_config="HAVE_MBRTOWC"' --cfg 'py_sys_config="HAVE_MREMAP"' --cfg 'py_sys_config="MODBUILT_NAMES"' --cfg 'py_sys_config="HAVE_ASINH"' --cfg 'py_sys_config="PY_LDFLAGS_NODIST"' --cfg 'py_sys_config="SRCDIRS"' --cfg 'py_sys_config="HAVE_STDLIB_H"' --cfg 'py_sys_config="HAVE_INET_ATON"' --cfg 'py_sys_config="HAVE_READV"' --cfg 'py_sys_config="STDC_HEADERS"' --cfg 'py_sys_config="HAVE_SETUID"' --cfg 'py_sys_config="HAVE_MKNOD"' --cfg 'py_sys_config="HAVE_LINUX_NETLINK_H"' --cfg 'py_sys_config="EXT_SUFFIX"' --cfg 'py_sys_config="HAVE_RENAMEAT"' --cfg 'py_sys_config="HAVE_TIMEGM"' --cfg 'py_sys_config="HAVE_UTIMES"' --cfg 'py_sys_config="PY_COERCE_C_LOCALE"' --cfg 'py_sys_config="PY_SSL_DEFAULT_CIPHERS"' --cfg 'py_sys_config="HAVE_GETGROUPLIST"' --cfg 'py_sys_config="HAVE_POLL"' --cfg 'py_sys_config="PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT"' --cfg 'py_sys_config="LDLIBRARY"' --cfg 'py_sys_config="HAVE_TGAMMA"' --cfg 'py_sys_config="PY_STDMODULE_CFLAGS"' --cfg 'py_sys_config="HAVE_WCSCOLL"' --cfg 'py_sys_config="SIZEOF_DOUBLE"' --cfg 'py_sys_config="DFLAGS"' --cfg 'py_sys_config="HAVE_LINUX_CAN_RAW_H"' --cfg 'py_sys_config="HAVE_SEM_UNLINK"' --cfg 'py_sys_config="SYSLIBS"' --cfg 'py_sys_config="HAVE_STATVFS"' --cfg 'py_sys_config="HAVE_GETC_UNLOCKED"' --cfg 'py_sys_config="OPENSSL_INCLUDES"' --cfg 'py_sys_config="HAVE_SYS_SYSCALL_H"' --cfg 'py_sys_config="PYTHON_FOR_BUILD"' --cfg 'py_sys_config="HAVE_SETLOCALE"' --cfg 'py_sys_config="LIBRARY_OBJS"' --cfg 'py_sys_config="HAVE_FORK"' --cfg 'py_sys_config="EXTRA_CFLAGS"' --cfg 'py_sys_config="prefix"' --cfg 'py_sys_config="HAVE_GAI_STRERROR"' --cfg 'py_sys_config="PY_LDFLAGS"' --cfg 'py_sys_config="HAVE_PUTENV"' --cfg 'py_sys_config="HAVE_SETVBUF"' --cfg 'py_sys_config="HAVE_FSEEKO"' --cfg 'py_sys_config="HAVE_NET_IF_H"' --cfg 'py_sys_config="HAVE_TMPFILE"' --cfg 'py_sys_config="LDFLAGS_NODIST"' --cfg 'py_sys_config="DISTFILES"' --cfg 'py_sys_config="HAVE_GETPWENT"' --cfg 'py_sys_config="LDLIBRARYDIR"' --cfg 'py_sys_config="HAVE_SYS_UTSNAME_H"' --cfg 'py_sys_config="datarootdir"' --cfg 'py_sys_config="HAVE_HSTRERROR"' --cfg 'py_sys_config="HAVE_ATANH"' --cfg 'py_sys_config="LIBM"' --cfg 'py_sys_config="SIZEOF_FPOS_T"' --cfg 'py_sys_config="HAVE_SETITIMER"' --cfg 'py_sys_config="HAVE_STD_ATOMIC"' --cfg 'py_sys_config="HAVE_HTOLE64"' --cfg 'py_sys_config="CONFINCLUDEPY"' --cfg 'py_sys_config="HAVE_SETHOSTNAME"' --cfg 'py_sys_config="HAVE_LARGEFILE_SUPPORT"' --cfg 'py_sys_config="IO_OBJS"' --cfg 'py_sys_config="RESSRCDIR"' --cfg 'py_sys_config="LDSHARED"' --cfg 'py_sys_config="PY_CORE_LDFLAGS"' --cfg 'py_sys_config="UNIVERSALSDK"' --cfg 'py_sys_config="MODDISABLED_NAMES"' --cfg 'py_sys_config="CONFIGURE_CFLAGS"' --cfg 'py_sys_config="HAVE_HYPOT"' --cfg 'py_sys_config="OTHER_LIBTOOL_OPT"' --cfg 'py_sys_config="LIBRARY_OBJS_OMIT_FROZEN"' --cfg 'py_sys_config="WITH_DOC_STRINGS"' --cfg 'py_sys_config="LIBRARY"' --cfg 'py_sys_config="HAVE_ZLIB_COPY"' --cfg 'py_sys_config="HAVE_PTHREAD_H"' --cfg 'py_sys_config="HAVE_MMAP"' --cfg 'py_sys_config="INCLUDEDIR"' --cfg 'py_sys_config="HAVE_PATHCONF"' --cfg 'py_sys_config="PY_BUILTIN_MODULE_CFLAGS"' --cfg 'py_sys_config="LIBDIR"' --cfg 'py_sys_config="CONFIGFILES"' --cfg 'py_sys_config="DESTPATH"' --cfg 'py_sys_config="DTRACE_DEPS"' --cfg 'py_sys_config="HAVE_DECL_RTLD_LOCAL"' --cfg 'py_sys_config="SIZEOF_FLOAT"' --cfg 'py_sys_config="PYTHON_OBJS"' --cfg 'py_sys_config="INSTALL"' --cfg 'py_sys_config="GITVERSION"' --cfg 'py_sys_config="INSTALL_SHARED"' --cfg 'py_sys_config="HAVE_GETPGRP"' --cfg 'py_sys_config="COVERAGE_INFO"' --cfg 'py_sys_config="LINKFORSHARED"' --cfg 'py_sys_config="HAVE_SYS_EPOLL_H"' --cfg 'py_sys_config="MODLIBS"' --cfg 'py_sys_config="PGENOBJS"' --cfg 'py_sys_config="HAVE_SETREGID"' --cfg 'py_sys_config="HAVE_FUTIMESAT"' --cfg 'py_sys_config="HAVE_KILLPG"' --cfg 'py_sys_config="CONFIGURE_LDFLAGS_NODIST"' --cfg 'py_sys_config="HAVE_CLOCK"' --cfg 'py_sys_config="LOCALMODLIBS"' --cfg 'py_sys_config="HAVE_WRITEV"' --cfg 'py_sys_config="HAVE_IF_NAMEINDEX"' --cfg 'py_sys_config="HAVE_SYS_TIME_H"' --cfg 'py_sys_config="SOABI"' --cfg 'py_sys_config="LDVERSION"' --cfg 'py_sys_config="PGOBJS"' --cfg 'py_sys_config="HAVE_COPYSIGN"' --cfg 'py_sys_config="HAVE_MEMORY_H"' --cfg 'py_sys_config="SHLIB_SUFFIX"' --cfg 'py_sys_config="HAVE_SCHED_SETPARAM"' --cfg 'py_sys_config="LIBOBJS"' --cfg 'py_sys_config="SIZEOF_SIZE_T"' --cfg 'py_sys_config="HAVE_LSTAT"' --cfg 'py_sys_config="SGI_ABI"' --cfg 'py_sys_config="HAVE_SYS_SOCKET_H"' --cfg 'py_sys_config="HAVE_REALPATH"' --cfg 'py_sys_config="HAVE_LINUX_VM_SOCKETS_H"' --cfg 'py_sys_config="PY_CORE_CFLAGS"' --cfg 'py_sys_config="HAVE_CHROOT"' --cfg 'py_sys_config="HAVE_STDARG_PROTOTYPES"' --cfg 'py_sys_config="HAVE_KILL"' --cfg 'py_sys_config="HAVE_SYS_UN_H"' --cfg 'py_sys_config="LIBPC"' --cfg 'py_sys_config="HOST_GNU_TYPE"' --cfg 'py_sys_config="HAVE_SSIZE_T"' --cfg 'py_sys_config="PYTHONFRAMEWORKINSTALLDIR"' --cfg 'py_sys_config="HAVE_FTELLO"' --cfg 'py_sys_config="HAVE_SYS_PARAM_H"' --cfg 'py_sys_config="SITEPATH"' --cfg 'py_sys_config="HAVE_PTHREAD_KILL"' --cfg 'py_sys_config="ANDROID_API_LEVEL"' --cfg 'py_sys_config="HAVE_ALLOCA_H"' --cfg 'py_sys_config="HAVE_SCHED_SETSCHEDULER"' --cfg 'py_sys_config="HAVE_SYS_TIMES_H"' --cfg 'py_sys_config="DYNLOADFILE"' --cfg 'py_sys_config="HAVE_SYSCONF"' --cfg 'py_sys_config="HAVE_FLOCK"' --cfg 'py_sys_config="QUICKTESTOPTS"' --cfg 'py_sys_config="HAVE_SETPRIORITY"' --cfg 'py_sys_config="LDLAST"' --cfg 'py_sys_config="MODOBJS"' --cfg 'py_sys_config="MACHDEP"' --cfg 'py_sys_config="PROFILE_TASK"' --cfg 'py_sys_config="HAVE_DECL_ISNAN"' --cfg 'py_sys_config="MAKESETUP"' --cfg 'py_sys_config="HAVE_DUP2"' --cfg 'py_sys_config="HAVE_TMPNAM"' --cfg 'py_sys_config="STRICT_SYSV_CURSES"' --cfg 'py_sys_config="LIBFFI_INCLUDEDIR"' --cfg 'py_sys_config="LIBOBJDIR"' --cfg 'py_sys_config="srcdir"' --cfg 'py_sys_config="HAVE_SCHED_GET_PRIORITY_MAX"' --cfg 'py_sys_config="HAVE_CTERMID"' --cfg 'py_sys_config="HAVE_DIRENT_H"' --cfg 'py_sys_config="HAVE_GETITIMER"' --cfg 'py_sys_config="PYTHONPATH"' --cfg 'py_sys_config="SIZEOF_PTHREAD_T"' --cfg 'py_sys_config="TCLTK_LIBS"' --cfg 'py_sys_config="HAVE_DUP3"' --cfg 'py_sys_config="HAVE_PWRITEV"' --cfg 'py_sys_config="HAVE_SIGTIMEDWAIT"' --cfg 'py_sys_config="HAVE_LOCKF"' --cfg 'py_sys_config="HAVE_WCHAR_H"' --cfg 'py_sys_config="PY_CFLAGS"' --cfg 'py_sys_config="FILEMODE"' --cfg 'py_sys_config="SIZEOF_PTHREAD_KEY_T"' --cfg 'py_sys_config="DTRACE"' --cfg 'py_sys_config="HAVE_GETPID"' --cfg 'py_sys_config="HAVE_FCHDIR"' --cfg 'py_sys_config="XMLLIBSUBDIRS"' --cfg 'py_sys_config="LIBPL"' --cfg 'py_sys_config="GITTAG"' --cfg 'py_sys_config="HAVE_STAT_TV_NSEC"' --cfg 'py_sys_config="LIBDEST"' --cfg 'py_sys_config="HAVE_SOCKADDR_ALG"' --cfg 'py_sys_config="HAVE_GETTIMEOFDAY"' --cfg 'py_sys_config="HAVE_DIRENT_D_TYPE"' --cfg 'py_sys_config="SUBDIRSTOO"' --cfg 'py_sys_config="LLVM_PROF_FILE"' --cfg 'py_sys_config="HAVE_SYSEXITS_H"' --cfg 'py_sys_config="PARSER_OBJS"' --cfg 'py_sys_config="SIZEOF_LONG_DOUBLE"' --cfg 'py_sys_config="PYTHON_HEADERS"' --cfg 'py_sys_config="CONFIGURE_CFLAGS_NODIST"' --cfg 'py_sys_config="HAVE_ADDRINFO"' --cfg 'py_sys_config="HAVE_MKDIRAT"' --cfg 'py_sys_config="SIZEOF_LONG"' --cfg 'py_sys_config="HAVE_SCHED_RR_GET_INTERVAL"' --cfg 'py_sys_config="HAVE_STRUCT_STAT_ST_RDEV"' --cfg 'py_sys_config="HAVE_MAKEDEV"' --cfg 'py_sys_config="HAVE_GETPRIORITY"' --cfg 'py_sys_config="HAVE_GETGROUPS"' --cfg 'py_sys_config="HAVE_OPENAT"' --cfg 'py_sys_config="HAVE_FTRUNCATE"' --cfg 'py_sys_config="LIPO_32BIT_FLAGS"' --cfg 'py_sys_config="LIBC"' --cfg 'py_sys_config="HAVE_SETGID"' --cfg 'py_sys_config="HAVE_SETPGID"' --cfg 'py_sys_config="INSTALL_PROGRAM"' --cfg 'py_sys_config="HAVE_INITGROUPS"' --cfg 'py_sys_config="SRC_GDB_HOOKS"' --cfg 'py_sys_config="HAVE_GETRESGID"' --cfg 'py_sys_config="DISTDIRS"' --cfg 'py_sys_config="HAVE_SCHED_H"' --cfg 'py_sys_config="HAVE_ERFC"' --cfg 'py_sys_config="Py_ENABLE_SHARED"' --cfg 'py_sys_config="BASECPPFLAGS"' --cfg 'py_sys_config="BINLIBDEST"' --cfg 'py_sys_config="DESTDIRS"' --cfg 'py_sys_config="HAVE_FCHMOD"' --cfg 'py_sys_config="DIRMODE"' --cfg 'py_sys_config="EXEMODE"' --cfg 'py_sys_config="EXE"' --cfg 'py_sys_config="HAVE_SIGINTERRUPT"' --cfg 'py_sys_config="MAJOR_IN_SYSMACROS"' --cfg 'py_sys_config="HAVE_SEM_OPEN"' --cfg 'py_sys_config="HAVE_FPATHCONF"' --cfg 'py_sys_config="LDFLAGS"' --cfg 'py_sys_config="BASECFLAGS"' --cfg 'py_sys_config="HAVE_LINUX_TIPC_H"' --cfg 'py_sys_config="HAVE_SENDFILE"' --cfg 'py_sys_config="HAVE_WCSFTIME"' --cfg 'py_sys_config="LDCXXSHARED"' --cfg 'py_sys_config="LN"' --cfg 'py_sys_config="HAVE_FSYNC"' --cfg 'py_sys_config="PYTHON"' --cfg 'py_sys_config="PYTHON_FOR_REGEN"' --cfg 'py_sys_config="HAVE_ENDIAN_H"' --cfg 'py_sys_config="INSTALL_SCRIPT"' --cfg 'py_sys_config="HAVE_POLL_H"' --cfg 'py_sys_config="HAVE_STDINT_H"' --cfg 'py_sys_config="HAVE_FINITE"' --cfg 'py_sys_config="PYTHONFRAMEWORKPREFIX"' --cfg 'py_sys_config="exec_prefix"' --cfg 'py_sys_config="HAVE_ALARM"' --cfg 'py_sys_config="HAVE_SYS_SYSMACROS_H"' --cfg 'py_sys_config="DLLLIBRARY"' --cfg 'py_sys_config="HAVE_STRUCT_TM_TM_ZONE"' --cfg 'py_sys_config="ENABLE_IPV6"' --cfg 'py_sys_config="HAVE_SETREUID"' --cfg 'py_sys_config="HAVE_LOG1P"' --cfg 'py_sys_config="CXX"' --cfg 'py_sys_config="HAVE_LCHOWN"' --cfg 'py_sys_config="HAVE_SYS_STATVFS_H"' --cfg 'py_sys_config="OPT"' --cfg 'py_sys_config="SIZEOF_LONG_LONG"' --cfg 'py_sys_config="HAVE_UTIMENSAT"' --cfg 'py_sys_config="HAVE_SYNC"' --cfg 'py_sys_config="MACOSX_DEPLOYMENT_TARGET"' --cfg 'py_sys_config="PY_CPPFLAGS"' --cfg 'py_sys_config="HAVE_GAMMA"' --cfg 'py_sys_config="COVERAGE_REPORT_OPTIONS"' --cfg 'py_sys_config="HAVE_UTIME_H"' --cfg 'py_sys_config="HAVE_GRP_H"' --cfg 'py_sys_config="LINKCC"' --cfg 'py_sys_config="HAVE_MKTIME"' --cfg 'py_sys_config="DOUBLE_IS_LITTLE_ENDIAN_IEEE754"' --cfg 'py_sys_config="HAVE_LINUX_RANDOM_H"' --cfg 'py_sys_config="HAVE_SETSID"' --cfg 'py_sys_config="HAVE_SYS_ENDIAN_H"' --cfg 'py_sys_config="HAVE_PWRITE"' --cfg 'py_sys_config="HAVE_EXPM1"' --cfg 'py_sys_config="HAVE_NICE"' --cfg 'py_sys_config="HAVE_GETLOGIN"' --cfg 'py_sys_config="HAVE_UNAME"' --cfg 'py_sys_config="HAVE_SIGWAITINFO"' --cfg 'py_sys_config="HAVE_SOCKETPAIR"' --cfg 'py_sys_config="HAVE_ASM_TYPES_H"' --cfg 'py_sys_config="EXTRATESTOPTS"' --cfg 'py_sys_config="HAVE_PTY_H"' --cfg 'py_sys_config="HAVE_STRDUP"' --cfg 'py_sys_config="HAVE_DECL_ISFINITE"' --cfg 'py_sys_config="CFLAGSFORSHARED"' --cfg 'py_sys_config="HAVE_SIGPENDING"' --cfg 'py_sys_config="INCLUDEPY"' --cfg 'py_sys_config="HAVE_SETGROUPS"' --cfg 'py_sys_config="HAVE_PREADV"' --cfg 'py_sys_config="HAVE_WCSXFRM"' --cfg 'py_sys_config="HAVE_SYS_TYPES_H"' --cfg 'py_sys_config="MAINCC"' --cfg 'py_sys_config="HAVE_SETEGID"' --cfg 'py_sys_config="GNULD"' --cfg 'py_sys_config="HAVE_SYS_WAIT_H"' --cfg 'py_sys_config="HAVE_INET_PTON"' --cfg 'py_sys_config="HAVE_STRUCT_PASSWD_PW_GECOS"' --cfg 'py_sys_config="PARSER_HEADERS"' --cfg 'py_sys_config="HAVE_SCHED_SETAFFINITY"' --cfg 'py_sys_config="PGO_PROF_GEN_FLAG"' --cfg 'py_sys_config="HAVE_SOCKADDR_STORAGE"' --cfg 'py_sys_config="HAVE_UNISTD_H"' --cfg 'py_sys_config="HAVE_STRFTIME"' --cfg 'py_sys_config="HAVE_PRLIMIT"' --cfg 'py_sys_config="abs_builddir"' --cfg 'py_sys_config="TESTOPTS"' --cfg 'py_sys_config="HAVE_FUTIMENS"' --cfg 'py_sys_config="HAVE_GETNAMEINFO"' --cfg 'py_sys_config="ARFLAGS"' --cfg 'py_sys_config="HAVE_SIGWAIT"' --cfg 'py_sys_config="PGO_PROF_USE_FLAG"' --cfg 'py_sys_config="CFLAGS_ALIASING"' --cfg 'py_sys_config="HAVE_EXECV"' --cfg 'py_sys_config="NO_AS_NEEDED"' --cfg 'py_sys_config="HAVE_SYMLINKAT"' --cfg 'py_sys_config="PYTHONFRAMEWORKDIR"' --cfg 'py_sys_config="HAVE_GETSID"' --cfg 'py_sys_config="HAVE_STRUCT_STAT_ST_BLOCKS"' --cfg 'py_sys_config="OBJECT_OBJS"' --cfg 'py_sys_config="PGEN"' --cfg 'py_sys_config="TESTPATH"' --cfg 'py_sys_config="HAVE_STRUCT_PASSWD_PW_PASSWD"' --cfg 'py_sys_config="HAVE_LINK"' --cfg 'py_sys_config="HAVE_PAUSE"' --cfg 'py_sys_config="DIST"' --cfg 'py_sys_config="SIZEOF_WCHAR_T"' --cfg 'py_sys_config="HAVE_TCGETPGRP"' --cfg 'py_sys_config="HAVE_SYS_IOCTL_H"' --cfg 'py_sys_config="MULTIARCH"' --cfg 'py_sys_config="HAVE_TCSETPGRP"' --cfg 'py_sys_config="CPPFLAGS"' --cfg 'py_sys_config="HAVE_NETPACKET_PACKET_H"' --cfg 'py_sys_config="HAVE_DECL_RTLD_LAZY"' --cfg 'py_sys_config="TESTPYTHON"' --cfg 'py_sys_config="HAVE_LINUX_CAN_RAW_FD_FRAMES"' --cfg 'py_sys_config="WITH_DECIMAL_CONTEXTVAR"' --cfg 'py_sys_config="BLDLIBRARY"' --cfg 'py_sys_config="UNICODE_DEPS"' --cfg 'py_sys_config="MKDIR_P"' -l python3.7m` |
It's used to ensure that our ffi definitions match the structs generated by similar configuration on the C side, e.g. here: Line 12 in b710186
However, from searching the codebase it looks like the only ones we need are So I think we should probably change |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Really helpful, though we can do it more simply.
I can fix that! |
Bad news is, none of these variables are exposed in the |
Ah, blast, sorry I suggested a bad approach. 😢 Thanks for continuing to figure this out! |
No problem, at this point it's basically research and experimentation. Happens all the time. I wont have much time to attribute to this before the end of this coming week though. I might read through python's documentation during lost time to maybe find a better solution than header parsing. |
@Progdrasil |
That seems like a good idea to me! Wrt presence of working Python interpreter on host - I think you're right most host systems will have one. I also think we're ok to require one (as long as we document it)! |
I tried In [5]: with open(sysconfig.get_config_h_filename()) as f:
...: d = sysconfig.parse_config_h(f) but So I think
For Py_TRACE_REFS, COUNT_ALLOCS, and Py_REF_DEBUG: we should just set these variables if |
Following the discussion in PyO3#1077 this change allows the compilation script to load the configurations from a _sysconfigdata_ file in the library directory. This file is also provided on target systems in the same directory. At least on Manjaro Linux. Which could remove the need to run a python script at compile time for compiling the the host. I've also addressed the linking need for android in PyO3#1082.
9a41219
to
aef159d
Compare
So after more investigation, as for the placement of this file, it is hard coded when generated, so i can search for this placement explicitly and if not found search for the file at the top level. This might however be less accurate than simply searching recursively within the lib directory, as the platform nomenclature between rust and python might not match up 100% of the time. Let me know which you prefer @davidhewitt and @kngwyu |
I'm sorry I don't understand the context completely, but please choose a simpler way for finding the file. |
Python saves the file during build time at the path defined by: pybuilddir = 'build/lib.%s-%s' % (get_platform(), sys.version_info[:2]) This would be the rust equivalent of let pybuilddir = format!("build/lib.{}-{}-{}", env!("CARGO_CFG_TARGET_OS"), env!("CARGO_CFG_TARGET_ARCH"), minor) at least with my cross compiled version, I get the added However since were searching for the file in order to find the version, we have to search for all the versions that PyO3 supports. |
Actually I could just search for a directory which starts with |
Sorry what is 'the file'... is it different from |
The pybuilddir = 'build/lib.%s-%s' % (get_platform(), sys.version_info[:2]) where
And it seems when it is packaged for distribution, the {PREFIX}/_sysconfigdata*.py # distribution from package manager
{PREFIX}/build/lib.linux-x86_64-{MINOR}/_sysconfigdata*.py # if built from source for host
{PREFIX}/build/lib.{OS}(-{OS-KERNEL-RELEASE})?-{ARCH}-{MINOR}/_sysconfigdata*.py # if cross compiled, kernel release is only present on certain OS targets. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much for the continued iterations and improvements to this PR. I think we're getting close now!
A bunch of suggestions from me to help round things off.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really nice work, very pleased with this result! Thank you ❤️
We got some feedback from a user who found this branch solved their problem too, though sounded like they needed to tweak the PYO3_CROSS_LIB
dir slightly. (See my other comment.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice 👍 , though I have some minor concerns.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
7beb272
looks great!
I'm afraid there's one final edge case on my machine:
$ ls -l /usr/lib/python*/_sysconfigdata*
-rw-r--r-- 1 root root 24866 Apr 18 03:59 /usr/lib/python3.7/_sysconfigdata_m_linux_x86_64-linux-gnu.py
lrwxrwxrwx 1 root root 35 Apr 27 16:53 /usr/lib/python3.8/_sysconfigdata__linux_x86_64-linux-gnu.py -> _sysconfigdata__x86_64-linux-gnu.py
-rw-r--r-- 1 root root 25708 Apr 27 16:53 /usr/lib/python3.8/_sysconfigdata__x86_64-linux-gnu.py
With PYO3_CROSS_LIB_DIR=/usr/lib
and PYO3_PYTHON_VERSION=3.8
, the current commit will match both the 3.8
configdatas and bail. See my other comment for a suggestion.
(Not sure why the default python 3.8 install on ubuntu 20.04 is laid out like that. Alas!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks brilliant to me now, thanks so much for your dedication over many revisions! I'll wait a bit to merge just incase others have anything to say.
Thanks, it was a nice experience. That being said, having this in a crate of its own will help greatly in testing and debugging all this logic. |
Thank you! |
Following the discussion in #1077 this change allows the compilation
script to load the configurations from a
_sysconfigdata_*.py
filein the library directory.
This file is also provided on target systems in the same directory.
At least on Manjaro Linux.
Which could remove the need to run a python script at compile time
for compiling the the host.
I've also addressed the linking need for android in #1082.
The paths provided are now checked if they exist.
Future improvements
The build script subsequently grew quite a bit.
I will probably contribute to https://github.com/davidhewitt/python-config in order to refactor most of the python configuration logic there. It should also improve the debugability of the build script.
Another issue I encountered was when I miss-spelled the path to one of the cross-compilation paths, checking if the paths supplied are valid before even trying to get the configs would be a good simple improvement. This would also help consumers of the library when they inevitably make mistakes.