Static linking of Python library for programs not supported #95

kayhayen · 2018-08-07T21:52:41Z

This hasn't worked so far.

Currently the issue seems to be that Py_GetArgcArgv collides with the symbols in a "libpythonXY.a" coming from "Modules/main.c".

Wo do not use that, and cannot use it. So the values would be wrong, while there are Python modules with semi wide usage, which rely on these functions to work correctly.

The idea therefore is that we can find a "libpython.a" that would be used, and turn it into a (cached) form with "main.o" removed from the archive.

What will be needed is to statically compile a CPython from source. Do not use --enable-shared doing that. Then examine the resulting "libpython.a" and try to remove "main.o" and see if linking to a program then works with Nuitka without any duplicate symbols errors.

The way caching is done can be seen in "makeWindowsMingGW64Lib" which gets an appdirs provided directory "nuitka_cache" from the outside. So it would only do that once then.

This should be a simple task. I invite others to try it for themselves and create a PR to include in Nuitka.

niothiel · 2018-08-14T23:42:45Z

FWIW: I was able to get this solution to work, as I am a user of pyenv. Adding more context for documentation purposes.

Environment

pyenv version: 1.2.6
OSX version: 10.13.6
Python version: pyenv-installed 3.6.6
Nuitka version: 0.5.33rc4 (factory install as of Tue Aug 14 19:38:20 EDT 2018)
clang:

Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Steps taken

Install python statically via pyenv: pyenv install 3.6.6

Find all libpython references:

$ cd $(pyenv root)/versions/3.6.6
$ pwd
/Users/xxxxxx/.pyenv/versions/3.6.6
$ find . -name 'libpython*'
./lib/python3.6/config-3.6m-darwin/libpython3.6m.a
./lib/libpython3.6m.a

Remove main.o
ar -d lib/python3.6/config-3.6m-darwin/libpython3.6m.a main.o
NOTE: In my testing, only the config directory needs to have main.o removed, not the other one.
Try compiling a simple file: python -m nuitka hello.py
Execute file successfully:
```
$ ./hello.exe
Hello world!
```

Results

After compiling an internal program, it succeeds and yields an acceptable executable.

Hopefully the above information is a confirmation of your thoughts, @kayhayen. Perhaps someone can use this to submit a patch. :)

kayhayen · 2018-08-15T10:55:09Z

Indeed, if somebody were to create a PR, that would be great. Using statically linked libpython would actually make for some nice optimization potential all by itself for link time optimization, which then will make a lot of sense. So there is some benefit to be had.

kayhayen · 2018-10-01T09:21:55Z

On Windows this probably not ever going to work, but for Linux and friends it will be great.

kayhayen · 2018-10-03T15:40:13Z

Checked with pyenv upstream, they are not interested in solving the main.o presence in their library, which probably was worth a try.

The main problem will be finding out what library file python is going to resolve to, which more or less means to emulate ld search path behavior. That is not straightforward. Checking if it has the issue, and then applying ar would be.

feluxe · 2018-10-03T16:14:23Z

The main problem will be finding out what library file python is going to resolve to, which more or less means to emulate ld search path behavior. That is not straightforward. Checking if it has the issue, and then applying ar would be.

I'm not sure if this really is the same situation, but pyinstaller allows to select a libpythonXY.a, using the LD_LIBRARY_PATH environment variable.

Maybe nuitka could provide a --libpy option to allow specifying a directory to look for a libpythonXY.a?

I found this issue: pyinstaller/pyinstaller#3363. To me it reads like a similar problem?

Please excuse me if this is nonsense. I really don't know much about the internals of nuitka and pyinstaller. I just ran into similar looking problems with both tools and thought I should share this. :)

kayhayen · 2018-10-03T16:19:59Z

@feluxe The problem isn't how to specify another libpythonXY.a, the problem is to find it ahead of time so we can use it as input, and provide the corrected copy, after patching it. The last part is the easy part. And I don't want anything non-outoftheboxish of course.

The "trouble" is, we would assume a job Scons and the linker normally take care of, to locate this library. Nothing to really want to get into.

kayhayen · 2018-10-27T13:28:20Z

I actually did implement the searching for libpython.a to check out how it fares with MiniConda in terms of performance, as using the DLL has proven to cause some issues recently. And this has actually worked fine, except that "LTO" had to switched off, due to me not using a compatible compiler, where neither gcc-6, gcc-7, gcc-8 were. The middle one due to a bug in the bytecode reader, the others due to a different bytecode level.

But without LTO, this is probably not nearly as much fun. However, there are compilers installable in MiniConda, and using those, I expect it to work. So this might be approaching a solution. In general however, say pyenv, which seems to have a crap libpython.a that solution won't apply. But since AnaConda has wide availability, that should be great.

I am still figuring out, how to install the static library into an AnaConda, MiniConda comes with it, but it may only be about finding the right package.

kayhayen · 2018-10-30T13:08:47Z

So the LTO problem can likely be solved by using the Anaconda compiler that can be installed with some conda install commands. For Miniconda/Anaconda static linking is now attempted by default and ought to work on the factory branch. The LTO feature needs to be enabled with --lto and like I said, may not work, but I expect it to work once the defaulting to the conda compiler if installed is happening.

As for pyenv seems nobody it taking this on, despite repeated requests. I will just give up on that then, given that Anaconda is available everywhere anyway.

feluxe · 2018-11-10T10:52:23Z

I just came across pyenv/pyenv#65

If you set PYTHON_CONFIGURE_OPTS="--enable-shared", pyenv will give you:

$ ls -1 ~/.pyenv/versions/3.7.1/lib

libpython3.7m.so
libpython3.7m.so.1.0
libpython3.so
pkgconfig
python3.7

kayhayen · 2018-11-10T11:07:28Z

That should work better @feluxe or so I am thinking.

With current code on factory or maybe develop already, static linking is currently automatic for AnaConda only. so the bad libpython is avoided.

I have yet to make measurements, but as seen in #150 there are drawbacks to using "libpython.so" definitely, so for best performance this won't be advisable.

feluxe · 2018-11-10T13:50:27Z

As for pyenv seems nobody it taking this on, despite repeated requests. ...

... as seen in #150 there are drawbacks to using "libpython.so" definitely, so for best performance this won't be advisable.

Hmm, this is really unfortunate, IMHO.

There is a large amount of people who have their Python dev-environments build around pip/pipenv/pyenv, who wouldn't like to add conda to the tool stack.

Do you know where pyenv's shortcomings related to nuitka are discussed? An issue related to this on the pyenv issue tracker would be good place for nuitka user to increase awareness on the pyenv side. The way things are going, I'm sure nuitka will continue raising in popularity and become an integral part of the python ecosystem. At some point it will be too big for pyenv to ignore.

kayhayen · 2018-11-11T10:18:33Z

I totally agree @feluxe which is why I went to the bug tracker some time ago, found a comment that suggesting to ask in the gitter chat before raising an issue, and so I did. Without success.

kayhayen · 2018-12-05T17:22:41Z

This sort of works, but currently setting the CC of AnaConda is a tad bit too manual and needs documentation for Linux and MacOS to be usable generally. Removing factory tag as the code is now in develop and will be in next stable release.

jayvdb · 2019-10-03T18:30:36Z

The quick solution of removing main.o from the .a doesnt work in Python 3.7.3. The result is:

[  276s] scons: building `cli.dist/cli' because it doesn't exist
[  276s] gcc -o cli.dist/cli -fno-lto -z noexecstack -export-dynamic -s -Wl,-R,'$ORIGIN' -Wl,--disable-new-dtags -Wl,-b -Wl,binary -Wl,cli.build/__constants.bin -Wl,-b -Wl,elf32-i386 -Wl,-defsym -Wl,constant_bin=_binary_cli_build___constants_bin_start @"cli.build/@sources.tmp" -L/usr/lib -ldl -lm /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a -ldl -lpthread -lutil -lrt -lm
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pathconfig.o): in function `pathconfig_global_init':
[  276s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pathconfig.c:111: undefined reference to `_PyCoreConfig_Read'
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pathconfig.c:121: undefined reference to `_PyCoreConfig_Clear'
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pathconfig.c:125: undefined reference to `_PyCoreConfig_Clear'
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `_Py_InitializeCore':
[  276s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:850: undefined reference to `_PyCoreConfig_Copy'
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:851: undefined reference to `_PyCoreConfig_Read'
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:866: undefined reference to `_PyCoreConfig_Clear'
[  276s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `_Py_InitializeCore_impl':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:658: undefined reference to `_PyCoreConfig_SetGlobalConfig'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:714: undefined reference to `_PyCoreConfig_Copy'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `_Py_InitializeMainInterpreter':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:910: undefined reference to `_PyMainInterpreterConfig_Copy'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `new_interpreter':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:1397: undefined reference to `_PyCoreConfig_Copy'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:1400: undefined reference to `_PyMainInterpreterConfig_Copy'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `_Py_InitializeFromConfig':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:1001: undefined reference to `_Py_Initialize_ReadEnvVarsNoAlloc'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:1012: undefined reference to `_PyMainInterpreterConfig_Read'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:1016: undefined reference to `_PyMainInterpreterConfig_Clear'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `Py_InitializeEx':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:1038: undefined reference to `_PyCoreConfig_Clear'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pylifecycle.o): in function `_Py_Initialize_ReconfigureCore':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:606: undefined reference to `_PyCoreConfig_SetGlobalConfig'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pylifecycle.c:608: undefined reference to `_PyCoreConfig_Copy'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /usr/lib/python3.7/config-3.7m-i386-linux-gnu/libpython3.7m.a(pystate.o): in function `PyInterpreterState_Clear':
[  277s] /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pystate.c:201: undefined reference to `_PyCoreConfig_Clear'
[  277s] /usr/lib/gcc/i586-suse-linux/9/../../../../i586-suse-linux/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.7.3/Python/pystate.c:202: undefined reference to `_PyMainInterpreterConfig_Clear'
[  277s] collect2: error: ld returned 1 exit status
[  277s] scons: *** [cli.dist/cli] Error 1
[  277s] scons: building terminated because of errors.

Briefly looking through the 3.7 code, and the Python 3.8 code, it looks like a non-trivial task for Python 3.7 to create a new .o which includes parts of the original built main.o . However main.c has been split quite neatly in Python 3.8. Oddly they have still included the now tiny main.o into the .a - it now contains very little of use to anyone who is intending to build something which doesnt look and feel like the real CPython.

Anyway, when compiling against Python 3.8.0 rc1, I get the following with gcc:

[ 1223s] scons: building `cli.build/MetaPathBasedLoader.o' because it doesn't exist
[ 1223s] gcc -o cli.build/MetaPathBasedLoader.o -c -fvisibility=hidden -std=c11 -fwrapv -fpartial-inlining -fno-lto -ffile-prefix-map=cli.build/static_src=/usr/lib/python3.8/site-packages/nuitka/build/static_src -fno-var-tracking -O3 -pipe -D_NUITKA_STANDALONE -D_NUITKA_SYSFLAG_BYTES_WARNING=0 -D_NUITKA_SYSFLAG_NO_SITE=1 -D_NUITKA_SYSFLAG_VERBOSE=0 -D_NUITKA_SYSFLAG_UTF8=1 -D_NUITKA_SYSFLAG_OPTIMIZE=0 -D__NUITKA_NO_ASSERT__ -D_NUITKA_FROZEN=163 -D_NUITKA_MODULE_COUNT=1036 -D_NUITKA_EXE -I/usr/include/python3.8 -I/usr/PC -Icli.build -I/usr/lib/python3.8/site-packages/nuitka/build/include -I/usr/lib/python3.8/site-packages/nuitka/build/static_src cli.build/MetaPathBasedLoader.c
[ 1224s] cli.build/MetaPathBasedLoader.c: In function ‘callIntoShlibModule’:
[ 1224s] cli.build/MetaPathBasedLoader.c:430:50: error: dereferencing pointer to incomplete type ‘PyInterpreterState’ {aka ‘struct _is’}
[ 1224s]   430 |     int dlopenflags = PyThreadState_GET()->interp->dlopenflags;
[ 1224s]       |                                                  ^~
[ 1224s] scons: *** [cli.build/MetaPathBasedLoader.o] Error 1

From the relnotes it appears that Py_BUILD_CORE_MODULE needs to be defined, and possibly also Py_BUILD_CORE, in order to get access to these defs. The 3.8 relnotes touch on this.

I notice that Python 3.8 isnt setup in the test suite yet. Is anyone attempting it? Are there plans/motivation to support it soon, or is it an undesirable distraction at the moment?

jayvdb · 2019-10-04T06:47:39Z

In Python 3.8, Py_GetArgcArgv' is now in initconfig.o`, and removing those objects results in

[ 2144s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(obmalloc.o): in function `new_arena':
[ 2144s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Objects/obmalloc.c:1232: undefined reference to `Py_IgnoreEnvironmentFlag'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(unicodeobject.o): in function `init_fs_encoding':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Objects/unicodeobject.c:15799: undefined reference to `_Py_DumpPathConfig'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pathconfig.o): in function `config_init_module_search_paths':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pathconfig.c:239: undefined reference to `_PyWideStringList_Clear'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pathconfig.c:258: undefined reference to `PyWideStringList_Append'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pathconfig.o): in function `pathconfig_global_read':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pathconfig.c:436: undefined reference to `_PyConfig_InitCompatConfig'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pathconfig.c:439: undefined reference to `PyConfig_Read'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pathconfig.c:447: undefined reference to `PyConfig_Clear'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `preconfig_get_global_vars':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:471: undefined reference to `Py_IsolatedFlag'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:472: undefined reference to `Py_IgnoreEnvironmentFlag'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:473: undefined reference to `Py_UTF8Mode'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `_PyPreConfig_Read':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:816: undefined reference to `Py_UTF8Mode'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `precmdline_parse_cmdline':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:208: undefined reference to `PyWideStringList_Append'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `_PyArgv_AsWstrList':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:98: undefined reference to `_PyWideStringList_Clear'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:104: undefined reference to `_PyWideStringList_Copy'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `_PyPreCmdline_Clear':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:117: undefined reference to `_PyWideStringList_Clear'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:118: undefined reference to `_PyWideStringList_Clear'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `preconfig_set_global_vars':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:497: undefined reference to `Py_IsolatedFlag'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:498: undefined reference to `Py_IgnoreEnvironmentFlag'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:502: undefined reference to `Py_UTF8Mode'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `_PyPreCmdline_SetConfig':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:165: undefined reference to `_PyWideStringList_Extend'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(preconfig.o): in function `_PyArgv_AsWstrList':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/preconfig.c:90: undefined reference to `_PyWideStringList_Clear'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `Py_InitializeEx':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:1069: undefined reference to `_PyConfig_InitCompatConfig'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `pyinit_core':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:846: undefined reference to `_PyConfig_InitCompatConfig'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:848: undefined reference to `_PyConfig_Copy'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:853: undefined reference to `PyConfig_Read'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `pyinit_config':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:668: undefined reference to `_PyConfig_Write'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `pycore_init_runtime':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:492: undefined reference to `_PyConfig_Write'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `pycore_create_interpreter':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:529: undefined reference to `_PyConfig_Copy'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `pyinit_core':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:869: undefined reference to `PyConfig_Clear'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `init_sys_streams':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:1969: undefined reference to `_Py_ClearStandardStreamEncoding'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `new_interpreter':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:1441: undefined reference to `_PyConfig_Copy'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `pyinit_core_reconfigure':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:466: undefined reference to `_PyConfig_Write'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:468: undefined reference to `_PyConfig_Copy'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `_Py_ReconfigureMainInterpreter':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:882: undefined reference to `_PyWideStringList_AsList'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `init_sys_streams':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:1969: undefined reference to `_Py_ClearStandardStreamEncoding'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:1969: undefined reference to `_Py_ClearStandardStreamEncoding'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pylifecycle.o): in function `Py_FdIsInteractive':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pylifecycle.c:2354: undefined reference to `Py_InteractiveFlag'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pystate.o): in function `PyInterpreterState_New':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pystate.c:209: undefined reference to `PyConfig_InitPythonConfig'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(pystate.o): in function `_PyInterpreterState_Clear':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/pystate.c:269: undefined reference to `PyConfig_Clear'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(sysmodule.o): in function `_PySys_ReadPreinitWarnOptions':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:2080: undefined reference to `PyWideStringList_Append'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(sysmodule.o): in function `_PySys_ReadPreinitXOptions':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:2098: undefined reference to `PyWideStringList_Append'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(sysmodule.o): in function `_PySys_InitMain':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:2881: undefined reference to `_PyWideStringList_AsList'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:2896: undefined reference to `_PyWideStringList_AsList'
[ 2145s] /usr/bin/ld: /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:2897: undefined reference to `_PyWideStringList_AsList'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(sysmodule.o): in function `_PySys_ClearAuditHooks':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:271: undefined reference to `Py_VerboseFlag'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(sysmodule.o): in function `sys_breakpointhook':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:419: undefined reference to `Py_IgnoreEnvironmentFlag'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(sysmodule.o): in function `PySys_SetArgv':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/./Python/sysmodule.c:3124: undefined reference to `Py_IsolatedFlag'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(fileutils.o): in function `encode_locale_ex':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/fileutils.c:743: undefined reference to `Py_UTF8Mode'
[ 2145s] /usr/bin/ld: /usr/lib/python3.8/config-3.8-i386-linux-gnu/libpython3.8.a(fileutils.o): in function `_Py_DecodeLocaleEx':
[ 2145s] /home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Python/fileutils.c:551: undefined reference to `Py_UTF8Mode'

kayhayen · 2019-10-04T08:33:26Z

@jayvdb Regarding 3.8, I have followed the betas, and updated Nuitka as necessary, there were a couple of bugs filed against CPython as a result of that.

I haven't looked at 3.8 for some time though.

The first goal is usually to get it to work with 3.7 test suite in a compatible fashion and add low hanging fruits of new features. I had that for beta3 or beta4, I forget. I am hopeful that their assignment expressions will be easy for Nuitka, since it always did those as part of its re-formulations, but I havent't take the time yet to look at it.

As stated in other channels, my family duties have made me fall back on Nuitka a lot recently, and that is probably only going to change this weekend. I have to make a plan how to prioritize things in catching up. Currently I have a lot of unread emails. But getting current 3.8 to work as before, doing the minor tweaks needed will be nice.

From experience I know that once 3.8 is released, I will be hammered with questions about it. Last time I manged to be there at release day. Since it's only rc1 I might still get there in time, or at least not that late. It's always hard to tell how heavy features really hard to follow.

Surely I will be posting this in a first "Nuitka this week" posting after a long time this weekend, or so I hope. :)

kayhayen · 2019-10-04T08:41:14Z

@jayvdb for static linking: That unusable main.o saga is a mystery to me, it renders things unusable. However, static linking and ideally LTO with Python is absolutely key to performance with Nuitka. My measurements with Anaconda on Linux speak volumes.

Going through DLL jump pads disables some optimization for unicode strings, making programs that use libpython way slower by default. In some instances, I am avoiding this by re-implementing the code in question. Painful, not always possible, and partial solution only.

Anaconda has a libpython.a that is LTO and then you hit issues about how the bytecode requires their gcc to be used, because gcc 6, 7, 8 (and I bet 9 too) are all incompatible with each other for LTO.

Not sure how good static linking without LTO is, but I believe it might be equally well. It's sad this is so under-supported by upstream there.

kayhayen · 2019-10-04T08:45:06Z

One day, maybe Nuitka should become a proper fork of CPython, it would make a bunch of things easier, although of course, out of the box usability will drop to zero in an instant.

jayvdb · 2019-10-09T07:12:38Z

I was able to get linking working on a regular Python 3.7m build (not with --disabled-shared) by doctoring the .a

ar x libpython3.7m.a main.o
strip --strip-symbol=Py_GetArgcArgv main.o
ar r libpython3.7m.a main.o

The generated cli dist still included lots of .so, including libpython3.7m.so, presumably because they were referred to by my app, and the standalone app core dumped while loading the .so it found.

Maybe the above doctoring will work for a simple app and the doctoring might be enough to the dockers at https://github.com/pypa/manylinux/ working for static linking of standalone (as opposed to module mode as discussed at #81).

I'm now roughly following https://github.com/indygreg/python-build-standalone to build a more inclusive static Python.

sztomi · 2020-02-19T10:14:36Z

Sharing my experience: I didn't have to do any of the ar dance with a pyenv-built python. The only thing that mattered is that python had to be built with --enable-static. If that was the case, nuitka preferred the static lib over the shared one. I think it would be nice if this could be explicitly selected. Had I wanted to link to a shared libpython, I would have had to recompile python with only a shared lib.

kayhayen · 2020-03-01T17:48:16Z

So static linking no longer fails. The solution is to make the colliding symbol weak. It is rarely needed anyway.

anki-code · 2020-03-17T13:22:36Z

I had an error for xxh project:

$ python -O -m nuitka --follow-imports xxh
/usr/bin/ld: /home/pc/miniconda3/lib/libpython3.7m.a(main.o): in function `Py_GetArgcArgv':
(.text.unlikely.Py_GetArgcArgv+0x0): multiple definition of `Py_GetArgcArgv'; ./static_src/MainProgram.o:MainProgram.c:(.text+0xc0): first defined here
collect2: error: ld returned 1 exit status
scons: *** [/home/pc/xxh/xxh/xxh.bin] Error 1

And this works for me now:

git clone -b develop https://github.com/Nuitka/Nuitka
cd Nuitka
python setup.py install
nuitka --follow-imports ../xxh

Thanks!

kayhayen · 2020-05-01T10:11:17Z

This is now in the latest pre-release, and will soon be released.

kayhayen · 2020-05-11T15:02:38Z

The release 0.6.8 was just made and contains the correction.

lermana · 2021-10-23T19:10:40Z

In case anyone else finds themself here, I ran into this issue in Ubuntu + Python 3.9 (Anaconda) and alleviated the issue through installing libpython-static like so:

RUN conda install -c conda-forge libpython-static

kayhayen · 2021-10-24T08:13:25Z

Yes, static linking was not supported at the time, but now is, and it warns about. I guess, it would be nice to have a PR for Anaconda that tells people to install just that to resolve it.

lermana · 2021-10-24T12:18:15Z

thanks @kayhayen ! I will see if I can submit such a PR in the next few days.

kayhayen added enhancement An improvement rather than a bug help wanted Please help with this, we think you can labels Aug 7, 2018

This was referenced Sep 3, 2018

Unable to install on Linux Mint #111

Closed

clang: error: linker command failed with exit code 1 #116

Closed

kayhayen mentioned this issue Sep 26, 2018

How to modify the environment of script compiled in non-standalone mode? #149

Closed

kayhayen mentioned this issue Oct 3, 2018

cloudpickle nuitka.NuitkaOptimizationError #159

Closed

kayhayen mentioned this issue Oct 17, 2018

Performance penalty for uncompiled code #177

Closed

kayhayen added factory For issues fixed in factory only and removed help wanted Please help with this, we think you can labels Nov 10, 2018

kayhayen self-assigned this Nov 10, 2018

kayhayen removed the factory For issues fixed in factory only label Dec 5, 2018

kayhayen mentioned this issue Jan 21, 2019

Error, call to 'install_name_tool' #51

Closed

kayhayen mentioned this issue May 31, 2019

multiple definition of Py_GetArgcArgv #409

Closed

kayhayen added the help wanted Please help with this, we think you can label Aug 10, 2019

kayhayen removed their assignment Aug 10, 2019

jayvdb mentioned this issue Oct 3, 2019

Question: How to compile without system library dependencies #538

Closed

jayvdb mentioned this issue Oct 4, 2019

nuitka fails to find Python headers msys2/MINGW-packages#4868

Closed

This was referenced Nov 30, 2019

Error in checking if Python belongs to a conda environment #580

Closed

Update check for conda envs to support Python > 3.6 (issue #580) #581

Merged

gioboske mentioned this issue Dec 28, 2019

Static linking for non-Windows systems and conda environment #595

Closed

kayhayen self-assigned this Mar 1, 2020

kayhayen added this to the 0.6.8 milestone Mar 1, 2020

kayhayen added the factory For issues fixed in factory only label Mar 1, 2020

kayhayen added develop For issues fixed in develop only and removed factory For issues fixed in factory only help wanted Please help with this, we think you can labels May 1, 2020

kayhayen removed the develop For issues fixed in develop only label May 11, 2020

kayhayen closed this as completed May 11, 2020

Nuitka locked and limited conversation to collaborators Oct 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Static linking of Python library for programs not supported #95

Static linking of Python library for programs not supported #95

kayhayen commented Aug 7, 2018

niothiel commented Aug 14, 2018

kayhayen commented Aug 15, 2018

kayhayen commented Oct 1, 2018

kayhayen commented Oct 3, 2018

feluxe commented Oct 3, 2018 •

edited

Loading

kayhayen commented Oct 3, 2018

kayhayen commented Oct 27, 2018

kayhayen commented Oct 30, 2018

feluxe commented Nov 10, 2018 •

edited

Loading

kayhayen commented Nov 10, 2018

feluxe commented Nov 10, 2018

kayhayen commented Nov 11, 2018

kayhayen commented Dec 5, 2018

jayvdb commented Oct 3, 2019 •

edited

Loading

jayvdb commented Oct 4, 2019

kayhayen commented Oct 4, 2019

kayhayen commented Oct 4, 2019

kayhayen commented Oct 4, 2019

jayvdb commented Oct 9, 2019

sztomi commented Feb 19, 2020

kayhayen commented Mar 1, 2020

anki-code commented Mar 17, 2020

kayhayen commented May 1, 2020

kayhayen commented May 11, 2020

lermana commented Oct 23, 2021

kayhayen commented Oct 24, 2021

lermana commented Oct 24, 2021

Static linking of Python library for programs not supported #95

Static linking of Python library for programs not supported #95

Comments

kayhayen commented Aug 7, 2018

niothiel commented Aug 14, 2018

Environment

Steps taken

Results

kayhayen commented Aug 15, 2018

kayhayen commented Oct 1, 2018

kayhayen commented Oct 3, 2018

feluxe commented Oct 3, 2018 • edited Loading

kayhayen commented Oct 3, 2018

kayhayen commented Oct 27, 2018

kayhayen commented Oct 30, 2018

feluxe commented Nov 10, 2018 • edited Loading

kayhayen commented Nov 10, 2018

feluxe commented Nov 10, 2018

kayhayen commented Nov 11, 2018

kayhayen commented Dec 5, 2018

jayvdb commented Oct 3, 2019 • edited Loading

jayvdb commented Oct 4, 2019

kayhayen commented Oct 4, 2019

kayhayen commented Oct 4, 2019

kayhayen commented Oct 4, 2019

jayvdb commented Oct 9, 2019

sztomi commented Feb 19, 2020

kayhayen commented Mar 1, 2020

anki-code commented Mar 17, 2020

kayhayen commented May 1, 2020

kayhayen commented May 11, 2020

lermana commented Oct 23, 2021

kayhayen commented Oct 24, 2021

lermana commented Oct 24, 2021

feluxe commented Oct 3, 2018 •

edited

Loading

feluxe commented Nov 10, 2018 •

edited

Loading

jayvdb commented Oct 3, 2019 •

edited

Loading