Packaged Pythons

[MaxSkaller]

BPO	10243
Nosy	@loewis, @ned-deily

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-11-04.19:22:36.625>
created_at = <Date 2010-10-30.06:46:29.375>
labels = ['expert-installation', 'performance']
title = 'Packaged Pythons'
updated_at = <Date 2010-11-04.21:04:34.427>
user = 'https://bugs.python.org/MaxSkaller'

bugs.python.org fields:

activity = <Date 2010-11-04.21:04:34.427>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2010-11-04.19:22:36.625>
closer = 'loewis'
components = ['Installation']
creation = <Date 2010-10-30.06:46:29.375>
creator = 'Max.Skaller'
dependencies = []
files = []
hgrepos = []
issue_num = 10243
keywords = []
message_count = 6.0
messages = ['119963', '119965', '120358', '120380', '120433', '120440']
nosy_count = 3.0
nosy_names = ['loewis', 'ned.deily', 'Max.Skaller']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue10243'
versions = ['Python 3.2']

[MaxSkaller]

Not sure if this is a bug or not. I am unable to find libpython.so for Python3 on either my Mac or Ubuntu. Perhaps this is a packaging fault, however some documentation in the Wiki suggests otherwise. It appears the builders have reverted to an archaic linkage pattern which I helped to get rid of (lets see, perhaps a decade ago?). Python 2.6, for example, does ship a shared library.

Python must be shipped with code linked as follows, I will explain below why, but first the list:

1. The mainline (C main() function) must be a stub which calls the real mainline, which is located in libpython.

2. The mainline MUST be compiled with C++ not C.

3. All extension libraries and add-ons to Python provided as shared libraries must be explicitly linked against libpython.

In particular it is NOT acceptable for any extension or shared library component to expect to find its symbols in the host application executable as the Wiki documentation seems to suggest (in a section which explains a bit of a workaround for OSX frameworks).

Now the reason it MUST be this way. First, any C++ code which is to be linked into an application, either statically, dynamically at load time, or under program control at run time, may require certain stuff to be in place (RTTI, streams, exception handling stuff, or whatever) which can only be put in place in the initialisation of the main application. Although the details are platform specific, it is simply not safe to permit C++ extension modules unless this step is taken.

Legacy or embedded systems may have to make do with a C mainline, and systems which don't support dynamic loading can also do without C++ compilation provided the pre-loaded extensions are all C.

On most major platforms, however, a C++ driver stub is required.

The second issue is also quite simple. It is quite incorrect in a modern computing environment to assume an *application* will be hosting the python interpreter directly. It is not only possible, it is in fact the case for my project, that were the Python interpreter to be called, it would from a shared library loaded under program control at run time.

Such an interpreter cannot be loaded at all if it isn't present in a library: it either has to be statically linked into the shared library making the call, with some ugly linker switches to make sure no symbols are dropped, or it has to be loaded dynamically. The latter case is the only viable option if the run time linker is unable to share symbols to a loaded application, and even if that is possible and can be arranged it is not likely to work so well if multiple shared libraries try to do it.

Similarly, even if you managed to load it somehow, any dynamically loaded extensions may or may not be able to find the symbols.

The ONLY reliable way to ensure extensions can find libpython symbols is to link them against libpython.

In fact, the mainline application should not only contain NO libpython symbols specifically to disable this wrong practice and break any bad extensions that rely on it, it should also, as explained, contain exactly one reference to libpython, which calls Python with argv, argc[] as if it were the mainline.

Just as motivation here: my product is an ultra-high performance programming language with a special construction to allow Python C-extension modules to be built. Its target is usually a shared library and that library is produced by first generating C++ and then compiling it with your native C++ compiler.

For a generated program to call Python interpreter, it HAS to be available in a shared library, and for any extension modules that interpreter loads, they HAVE to get their symbols from that shared library, and, if the generated program is itself a Python module, then if that module is to be loaded from any C extension, including itself or some other extension, it HAS to be linked against libpython which HAS to be loaded dynamically by the loaded.

The unfortunate downside of this is that it is NOT POSSIBLE to have a huge statically linked Python executable which just loads C extensions and nothing else happens. If you're loading any C extensions dynamically libpython must be loaded dynamically too.

Just to be clear: I can easily build it the way I want it but this will not solve my problem, which is to support clients who wish to use my product to generate high performance Python callable modules which will just work "out of the box" with existing Python code. In particular, replacing some slow modules with optimised ones would be more or less entirely transparent .. except that at the moment it could only work with Python 2.x since Python 3.x shipments don't seem to have any shared libpython included (and I just changed my compiler to support Python 3 modules instead of Python 2).

[loewis]

Python 2.6, for example, does ship a shared library.

That is not really the case. Python 2.6 ships in source form. It
builds with a libpython shared library only if you configure
with --enable-shared.

In particular it is NOT acceptable for any extension or shared
library component to expect to find its symbols in the host
application executable as the Wiki documentation seems to suggest (in
a section which explains a bit of a workaround for OSX frameworks).

I completely disagree with this evaluation. These statements are not
true in general, although they might be partially true for specific
operating systems and compilers.

Although the details are platform specific, it is simply
not safe to permit C++ extension modules unless this step is taken.

It certainly is safe on all platforms where Python does that,
i.e. primarily Linux, Windows, and OS X. Only archaic C++
implementations fail to support applications where main()
was not compiled with a C++ compiler.

On most major platforms, however, a C++ driver stub is required.

That is not true.

It is not only possible, it
is in fact the case for my project, that were the Python interpreter
to be called, it would from a shared library loaded under program
control at run time.

Python supports embedding of the interpreter quite well. You can
choose between linking Python as a static (.a) or dynamic library.
If you use a static library, extension modules would still pick
up Python symbols from the executable.

Such an interpreter cannot be loaded at all if it isn't present in a
library: it either has to be statically linked into the shared
library making the call, with some ugly linker switches to make sure
no symbols are dropped, or it has to be loaded dynamically.

The Python build process will *always* create a library that embedding
applications can link against.

For a generated program to call Python interpreter, it HAS to be
available in a shared library, and for any extension modules that
interpreter loads, they HAVE to get their symbols from that shared
library, and, if the generated program is itself a Python module,
then if that module is to be loaded from any C extension, including
itself or some other extension, it HAS to be linked against libpython
which HAS to be loaded dynamically by the loaded.

If that's your requirement, make sure you build against a Python
installation that was itself built with --enable-shared.

The unfortunate downside of this is that it is NOT POSSIBLE to have a
huge statically linked Python executable which just loads C
extensions and nothing else happens. If you're loading any C
extensions dynamically libpython must be loaded dynamically too.

That depends on the operating system. On Linux, it would be possible
to link the entire Python interpreter in your language's shared library,
and extension modules would still find their symbols.

Just to be clear: I can easily build it the way I want it but this
will not solve my problem, which is to support clients who wish to
use my product to generate high performance Python callable modules
which will just work "out of the box" with existing Python code. In
particular, replacing some slow modules with optimised ones would be
more or less entirely transparent .. except that at the moment it
could only work with Python 2.x since Python 3.x shipments don't seem
to have any shared libpython included (and I just changed my compiler
to support Python 3 modules instead of Python 2).

One solution could be to ship libpythonXY.so yourself, along with your
the rest of the binaries that you provide. Or else do as I propose
above: link the Python interpreter statically into your shared library.

I fail to see the bug in this report.

[MaxSkaller]

On Sat, Oct 30, 2010 at 6:40 PM, Martin v. Löwis <report@bugs.python.org>wrote:

It may be there is none. You need to read the bit where I explain that I am
not building Python, I'm grabbing pre-made packages, for OSX and for Ubuntu.

The problem is that these packages don't seem to supply a dynamic link
version.

My problem cannot be solved by telling me I can build Python myself with
the --enable-shared switch, because I am not the client. I am a vendor
supplying a tool that can generate Python shared libraries which cannot run
unless the CLIENT has a shared library version of libpython. So you're
telling me to tell THEM to build Python with --enable-shared switch which is
out of the question for many of them, who may, for example, be programmers
working in a shop where they do not have the ability to change the system
installed by their system admin.

So the problem is that the *packagers* are not supplying the dynamic lib.

Surely that is not the Python dev's issue directly, but it IS an issue the
Python dev's can do something about, by talking to the packagers.

Anyhow, I will give up. I can't test the feature of the compiler I have
implemented
because I don't have a libpython.so and I have no intention of building one,
because I can't expect all those Python users out there to do it either.

It seems you really don't understand the requirements for dynamic linking:
my application code is exclusively in a dlopen()d shared library, so if it
is
used AS a python module or it wants itself to USE a Python extension
module OR Python interpreter itself, it cannot do so.

The top level application is a fixed mainline which does not include
libpython.a or any such symbols.

It's a basic design principle, Meyer called it "the principle of explicit
interfaces"
which means: if you depend on something make the dependency explicit.

Extension modules which do not *explicitly* link against libpython break
this rule.

[ned-deily]

For what it's worth, the python.org installers for Mac OS X do include a libpython shared library. As of Python 2.7 (and 3.2), the installer includes a symlink to make it easier to find:

$ cd /Library/Frameworks/Python.framework/Versions/2.7/lib
$ ls -l libpython2.7.dylib
$ lrwxr-xr-x  1 root  admin  9 Oct 27 20:46 libpython2.7.dylib@ -> ../Python
$ file ../Python
../Python: Mach-O universal binary with 2 architectures
../Python (for architecture ppc):	Mach-O dynamically linked shared library ppc
../Python (for architecture i386):	Mach-O dynamically linked shared library i386
$ cd /Library/Frameworks/Python.framework/Versions/2.6
$ file Python
Python: Mach-O universal binary with 2 architectures
Python (for architecture ppc):	Mach-O dynamically linked shared library ppc
Python (for architecture i386):	Mach-O dynamically linked shared library i386

In Debian and Ubuntu releases, the shared library is put into its own optionally installed package, libpythonx.x.

[MaxSkaller]

On Thu, Nov 4, 2010 at 5:19 PM, Ned Deily <report@bugs.python.org> wrote:

Ned Deily <nad@acm.org> added the comment:

For what it's worth, the python.org installers for Mac OS X do include a
libpython shared library. As of Python 2.7 (and 3.2), the installer
includes a symlink to make it easier to find:

$ cd /Library/Frameworks/Python.framework/Versions/2.7/lib
$ ls -l libpython2.7.dylib

Ok.. so why is it called Python instead of Python.dylib?

/Library/Frameworks/Python.framework>file Python
Python: broken symbolic link to Versions/Current/Python

/Library/Frameworks/Python.framework/Versions/3.1>file Python
Python: Mach-O universal binary with 2 architectures
Python (for architecture ppc): Mach-O dynamically linked shared library ppc
Python (for architecture i386): Mach-O dynamically linked shared library
i386

Hmm .. i386? Oh dear, I'm running Snow Leopard and I generate 64 bit code.

[ned-deily]

It's called Python because that's the way frameworks on OS X are generally structured: the shared library file has the same name as the framework. The Apple developer docs have lots of information on frameworks.

Prior to 2.7 and the upcoming 3.2 releases, python.org installers (including for 3.1) have been 32-bit only. Starting with 2.7 and the latest 3.2 alpha, there are two OS X installer downloads for each release, one 32-bit only (10.3 and later) and the other 32-bit/64-bit for use on 10.6. Also the Apple-supplied Python 2.6 in 10.6 is 32-bit/64-bit. Otherwise, if you need 64-bit for earlier releases, you will need to look elsewhere. For example, some of the third-party open source distributors for OS X like MacPorts support 64-bit builds of Python 2.6 and 3.1. Or you will need to build from scratch.