Allow building python as shared library

[ondrap]

BPO	527027
Nosy	@mwhudson, @gvanrossum, @loewis
Files	ppython.diff pp2.diff: Patch against CVS python

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2002-03-29.16:30:27.000>
created_at = <Date 2002-03-07.16:45:13.000>
labels = ['build']
title = 'Allow building python as shared library'
updated_at = <Date 2002-03-29.16:30:27.000>
user = 'https://bugs.python.org/ondrap'

bugs.python.org fields:

activity = <Date 2002-03-29.16:30:27.000>
actor = 'loewis'
assignee = 'loewis'
closed = True
closed_date = None
closer = None
components = ['Build']
creation = <Date 2002-03-07.16:45:13.000>
creator = 'ondrap'
dependencies = []
files = ['4035', '4036']
hgrepos = []
issue_num = 527027
keywords = ['patch']
message_count = 19.0
messages = ['39167', '39168', '39169', '39170', '39171', '39172', '39173', '39174', '39175', '39176', '39177', '39178', '39179', '39180', '39181', '39182', '39183', '39184', '39185']
nosy_count = 5.0
nosy_names = ['mwh', 'gvanrossum', 'loewis', 'ondrap', 'smurf']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue527027'
versions = ['Python 2.3']

[ondrap]

This patch allows building python as a shared library.

• enables building shared python with
  '--enable-shared-python' configuration option

• builds the file '.so' by default and changes the
  name on installation, so it is currently enabled on
  linux to be '0.0', but this can be easily changed

• tested on linux, solaris(gcc), tru64(cc) and HP-UX
  11.0(aCC). It produces the library using LDSHARED -o,
  while some architectures that were already building
  shared, used different algorithm. I'm not sure if it
  didn't break them (someone should check DGUX and BeOS).
  It also makes building shared library disabled by
  default, while these architectures had it enabled.

• it rectifies a small problem on solaris2.8, that
  makes double inclusion of thread.o (this produces error
  on 'ld' for shared library).

[gvanrossum]

Logged In: YES
user_id=6380

Could you submit the thread.o double inclusion patch
separately? It's much less controversial.

I like the idea of building Python as a shared lib, but I'm
hesitant to add more code to an already super complex area
of the configuration and build process.

I need more reviewers. Maybe the submitter can get some
other developers to comment?

P.S. it would be better if you used the current CVS or at
least the final 2.2 release as a basis for your patch.

[loewis]

Logged In: YES
user_id=21627

As the first issue, I'd like to clarify ownership of this
code. This is the same patch as bpo-497102, AFAICT, but
contributed by a different submitter. So who wrote created
that code originally?

The same comments that I made to bpo-497102 apply to this patch
as well: why 0.0; please no unrelated changes (Hurd); why
create both pic and non-pic objects; please no
compiler-specific flags in the makefile; why LD_PRELOAD.

[smurf]

Logged In: YES
user_id=10327

IMHO this patch has a couple of problems.

The main one is that GNU configure has standard options for enabling shared library support, --enable/disable-shared/static. They should be used!

The other is that it's Linux-only. Shared library support tends to work well, for varying definitions of "well" anyway, on lots of platforms, but you really need to use libtool for it. That would also get rid of the LD_PRELOAD, since that'd be encapsulated by libtool.

It's a rather bigger job to convert something like Python to libtool properly instead of hacking the Makefile a bit, and the build will definitely get somewhat slower as a result, BUT if we agree that a shared Python library is a good idea (i think it is!), the work is definitely worth doing.

[ondrap]

Logged In: YES
user_id=88611

Sorry, I've been inspired by the former patch and I have
mistakenly included it here. My patch doesn't use LD_PRELOAD
and creates the .a with -fPIC, so it is compatibile with
other makes (not only GNU). I'll try to learn libttool and
and try to do it that way though.

[loewis]

Logged In: YES
user_id=21627

While I agree on the "not Linux only" and "use standard
configure options" comments; I completely disagree on
libtool - only over my dead body. libtool is broken, and it
is a good thing that Python configure knows the compiler
command line options on its own.

[gvanrossum]

Logged In: YES
user_id=6380

libtool sucks. Case closed.

[ondrap]

Logged In: YES
user_id=88611

Ok, so no libtool. Did I get correctly, that you want:
--enable-shared/--enable-static instead of
--enable-shared-python, --disable-shared-python

• Do you agree with the way it is done in the patch
  (ppython.diff) or do you propose another way?

[loewis]

Logged In: YES
user_id=21627

Yes, that is all right. The approach, in general, is also
good, but please review my comments to bpo-497102.

Also, I still like to get a clarification as to who is the
author of this code.

[mwhudson]

Logged In: YES
user_id=6656

This ain't gonna happen on the 2.2.x branch, so changing group.

[ondrap]

Logged In: YES
user_id=88611

As far as I can see, the problems are:
relocation of binary/library path (this is solved by
adding -R to LDSHARED depending on platform)
SOVERSION - some systems like it, some do not. If you do
SOVERSION, you must create a link to the proper version in
the installation phase. IMO we can just avoid versioning
at all and let the distribution builders do it themselves.
The other way is to attach full version of python as
SOVERSION (e.g. 2.1.1 -> libpython2.1.so.2.1.1).

I'm the author of the patch (ppython.diff). I'm not the
author of the file dynamic.diff, I have included it here
by accident and if it is possible to delete it from this
page, it should be done.

[loewis]

Logged In: YES
user_id=21627

The patch looks quite good. There are a number of remaining
issues that need to be resolved, though:

• please regenerate the patch against the current CVS. As
  is, it fails to apply; parts of it are already in the CVS
  (the thr_create changes)

• I think the SOVERSION should be 1.0, atleast initially:
  for most Python releases, there will be only a single
  release of the shared library, which should be named 1.0.

• Why do you think that no rpath is needed on Linux? It is
  not needed if prefix is /usr, and on many installations, it
  is also not needed if prefix is /usr/local. For all other
  configurations, you still need a rpath on Linux.

• IMO, there could be a default case, assuming SysV-ish
  configurations.

[smurf]

Logged In: YES
user_id=10327

A SOVERSION of 0.0 makes perfect sense for the CVS head.

Release versions should probably use 1.0.

I don't quite know, though, if builds from CVS should keep a fixed SOVERSION -- after all, the API can change. One idea would be to use the tip version number of Doc/api/api.tex, i.e. libpython2.3.so.0.154 or libpython2.3.154.so.0.0.
That way, installing a newer CVS version won't instantly beak everything people have built with it.

[loewis]

Logged In: YES
user_id=21627

The CVS version will usually use a completely different
library name (e.g. libpython23.so), so there will be no
conflicts with prior versions.

[smurf]

Logged In: YES
user_id=10327

This is exactly the problem -- if today's libpython23.so replaces last week's libpython23.so, then everything I built during the last week is going to break if the ABI changes.

That's why I think that incorporating the version number from api.tex is a good idea -- call me an optimmist, but I think that any change will be documented. ;-)

This kind of problem is NOT pretty. I went through it a few years ago when the GNU libc transitioned to versioned linking. It managed to cause a LOT of almost-intractable incompatibilities during that time, and I don't care at all to repeat that experience with Python. :-(

[loewis]

Logged In: YES
user_id=21627

The API version is maintained in modsupport.h:API_VERSION.

I'm personally not concerned about breakage of API during
the development of a new release. Absolutely no breakage
should occur in maintenance releases. After all, a
maintenance will replace pythonxy.dll on Windows with no
protection against API breakage, thus, it is a bug if the
API changes in a maintenace release.

[ondrap]

Logged In: YES
user_id=88611

I have rebuilt the patch against CVS

• --enable-shared instead of --enable-shared-python
• sets rpath on Linux and Tru64 too
• I didn't change the SOVERSION stuff. I think we should
  come to a conclusion with versioning first. BTW: am I
  correct that make install should create the symlink .sl ->
  .sl.1.0 when we use versioning?
• this patch may break BeOS and DgUX. I think somone with
  access to these platforms should test it (he should use
  --enable-shared, as this patch changes the default behavior
  to --disable-shared for all platforms).

[loewis]

Logged In: YES
user_id=21627

I think the remaining issues are shallow only: Few users
will care about --enable-shared on BeOS and DG/UX; those
will hopefully contribute patches. Likewise, for .sl
libraries - I don't know HP-UX shared linking well enough to
determine whether it supports library versions.

[loewis]

Logged In: YES
user_id=21627

Thanks for the patch, committed as

Makefile.pre.in 1.78
README 1.143
configure 1.292
configure.in 1.302
ACKS 1.164
NEWS 1.372

I hope you'll stay around to deal with the bug reports for
this feature :-)