Support the *disabled* marker in Setup files

[tpetazzoni]

BPO	20210
Nosy	@malemburg, @loewis, @Yhg1s, @doko42, @pitrou, @vstinner, @merwok, @skrah, @xdegaye, @koobs, @tpetazzoni
PRs	bpo-20210: support the *disabled* marker in Setup files #132
Files	0001-Add-infrastructure-to-disable-the-build-of-certain-e.patch 0002-Add-an-option-to-disable-installation-of-test-module.patch 0003-Add-an-option-to-disable-pydoc.patch 0004-Add-an-option-to-disable-lib2to3.patch 0005-Add-option-to-disable-the-sqlite3-module.patch 0006-Add-an-option-to-disable-the-tk-module.patch 0007-Add-an-option-to-disable-the-curses-module.patch 0008-Add-an-option-to-disable-expat.patch 0009-Add-an-option-to-disable-CJK-codecs.patch 0010-Add-an-option-to-disable-NIS.patch 0011-Add-an-option-to-disable-unicodedata.patch 0012-Add-an-option-to-disable-IDLE.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-27.16:28:37.022>
created_at = <Date 2014-01-09.22:52:42.776>
labels = ['type-feature', '3.7', 'build']
title = 'Support the *disabled* marker in Setup files'
updated_at = <Date 2017-10-21.14:13:11.767>
user = 'https://github.com/tpetazzoni'

bugs.python.org fields:

activity = <Date 2017-10-21.14:13:11.767>
actor = 'xdegaye'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-27.16:28:37.022>
closer = 'xdegaye'
components = ['Build']
creation = <Date 2014-01-09.22:52:42.776>
creator = 'thomas-petazzoni'
dependencies = []
files = ['33380', '33381', '33382', '33383', '33384', '33385', '33386', '33387', '33388', '33389', '33390', '33391']
hgrepos = []
issue_num = 20210
keywords = ['patch']
message_count = 34.0
messages = ['207802', '214742', '214746', '214747', '214749', '214751', '214782', '214792', '214809', '214828', '214883', '214884', '214885', '214888', '214892', '214896', '236968', '237000', '237114', '242156', '263138', '267346', '267545', '267555', '267566', '287516', '287518', '287520', '287522', '287534', '287537', '287930', '294174', '294600']
nosy_count = 14.0
nosy_names = ['lemburg', 'loewis', 'twouters', 'doko', 'pitrou', 'vstinner', 'eric.araujo', 'Arfrever', 'skrah', 'xdegaye', 'koobs', 'thomas-petazzoni', 'wdv4758h', 'krasimir_vanev']
pr_nums = ['132']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20210'
versions = ['Python 3.7']

[tpetazzoni]

In the context of space-constrained embedded Linux systems, installing the entire set of Python modules and extensions is not necessarily desirable. For example, all the test modules, as well as certain extensions requiring third-parties libraries are often unnecessary, and uselessly consume precious storage on the embedded Linux system.

While we could certainly remove these undesired modules and extensions manually, it is much more convenient to have configuration options to selectively enable and disable them. Another very strong benefit of having configuration options is that we can actually *disable* the build of these unneeded modules and extensions, therefore saving a lot of build time, which is very nice when you're repeatedly cross-compiling an entire embedded Linux system.

The proposed set of patches add several --enable-<foo>/--disable-<foo> options to enable/disable certain Python modules and extensions. These patches have been part of Buildroot, an embedded Linux build system (used for example by Google, and many embedded processor vendors, as well as a huge number of embedded system makers) for a while, and are useful to all our users using Python on their embedded Linux systems. Instead of carrying them around, we would like to have them merged in upstream Python.

Of course, we are definitely open to discussion on the approach to take to implement this configurability, and I'm ready to rework the patches according to the comments received here.

Thanks!

[pitrou]

I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?

[tpetazzoni]

No, because it's not simply about the size of the installed Python standard library: it's also about the number of dependencies to build before being able to build Python. For example, a normal Python installation requires OpenSSL, libncurses, and lots of other things. On many embedded systems, those are not needed.

[pitrou]

For example, a normal Python installation requires OpenSSL,
libncurses, and lots of other things.

Not really. If some development libraries are not available, Python should still install fine without the corresponding modules.

[merwok]

The main issue with the proposed changes is that it redefines what “the Python standard library” is. Right now, users can mostly expect modules listed in the official Python docs to be available in their installation, regardless of how they got their Python.

I say “mostly” because a distributor may exclude tests from a binay package, split some extensions modules like _tkinter and _lzma in other packages, etc. A big source of pain in the past was distributors splitting distutils, but I think they all stopped.

(A related issue is that some distributors backport many bug fixes and sometimes features, which is a pain when you think you’ve run your tests with “Python X.Y.Z” and it was actually “X.Y.Z+some-patches”.)

[tpetazzoni]

But this expectation is not true: if dependencies are not available, Python silently disables the build of certain modules. So this story of making the standard library always has all the modules does not really stand.

[doko42]

But this expectation is not true: if dependencies are not available,
> Python silently disables the build of certain modules. So this story
of making the standard library always has all the modules does not really stand.

I think this one is a valid concern. Did run into it myself, because a final
package was built on another kernel and disabled some semaphore stuff by accident.

None of the extensions above are built as builtins by default, so you can always
prune these after the build. What I think is needed is a mode which breaks the
build when some extensions are not built. Whether this should be the default or
not, ... but it would give you a chance for a deterministic build. And the
build system already tells you at the end of the extension builds that some
extensions didn't build which were expected to build.

[koobs]

These are a good step toward bringing the otherwise neglected Python build system back to the real world in terms of standard functionality, and will among other things, make life an absolute pleasure for downstreams and users alike.

"User-Serviceable" options are expected (in particular in autotools-based build systems), *not* a luxury, and have been missing from the start.

A complicated build system is not a function its feature-set or flexibility, but of the quality of its evolution.

There is also a distinction between the ability to customise the options of a build, and the defaults of those options. "Will no longer be a standard library" is a straw man.

These patches present only as a user-configurable extension to otherwise statically defined configurations that must be patched manually to modify. This is painful.

With my downstream (FreeBSD) porter & consumer-and-hacker-of-Python-build-mechanics hat on, I'd like to see these and more 'options' out-of-the-box.

[pitrou]

A complicated build system is not a function its feature-set or
flexibility, but of the quality of its evolution.

Certainly, but that doesn't change the concrete issue: we have a complicated build system that these patches will make more complicated.

[BreamoreBoy]

I certainly like the principle. Does this need a wider audience, python-dev maybe?

[Arfrever]

If it is planned to support BSD make, then partial rewrite of patches will be needed.

Example of syntax of GNU make:
ifeq (something,something)
…
endif

Example of syntax of FreeBSD make:
.if ${variable}==something
…
.endif

[Arfrever]

According to koobs, building of CPython with FreeBSD make works at least with -j1 (and sporadically fails with higher value).

[koobs]

More precisely:

Python 3.3 fails at anything > -j1 (switching to gmake makes this go away)
Python 3.4 has not failed up to -j8 (with bsd make)

[vstinner]

Antoine Pitrou wrote:
"I don't really like the idea of complicating our build tools even more. Can't you simply prune the install tree yourself?"

In the embedded world, the (cross) compilation process is very complex and slow. Being able to disable features makes this task simpler.

Éric Araujo wrote:
"The main issue with the proposed changes is that it redefines what “the Python standard library” is."

I disagree. It's a common practice that a vendor gives the user the choice to enable or disable some features. On Gentoo and FreeBSD, you can disable features like IPv6 or shared memory on some packages. I don't think that it's currently possible on Python, but I would not be surprised to be able to enable or disable some features.

Éric Araujo wrote:
"Right now, users can mostly expect modules listed in the official Python docs to be available in their installation, regardless of how they got their Python."

How you get Python matters :-) Python documentations describes the vanilla flavor distributed at python.org. But the Python license allows to strip some features without changing the name of the Python.

--

0002-Add-an-option-to-disable-installation-of-test-module.patch is interested. I never understand why Python installs its test suite. Who use this test suite installed on the system? Maybe the packager of the module to test Python. Ok, but the test suite can then be removed.

I like the overall approach, by individual patches may be discussed. For example, 0008-Add-an-option-to-disable-expat.patch breaks Python XML modules. Are they still be installed? I mean the modules implemented in Python and relying on the expat Python module.

The changes should be be documented somewhere. In the Python documentation, or at least in the "devguide".

[pitrou]

Antoine Pitrou wrote:
"I don't really like the idea of complicating our build tools even
more. Can't you simply prune the install tree yourself?"

In the embedded world, the (cross) compilation process is very complex
and slow. Being able to disable features makes this task simpler.

That's not really the point. The question is why we should have to
maintain this ourselves. It is easy for interested people to maintain
their own forks, especially when *removing* stuff.

For the record, we don't have a single cross-compiling buildbot: it
isn't a supported setup.

0002-Add-an-option-to-disable-installation-of-test-module.patch is
interested. I never understand why Python installs its test suite. Who
use this test suite installed on the system? Maybe the packager of the
module to test Python. Ok, but the test suite can then be removed.

How else do you want to test that your Python installation works, other
than running the test suite?

[loewis]

I'm +1 on the general idea, but -1 on the implementation strategy used.

Instead of coming up with configure options for selected (apparently problematic) modules, I'd like to see a solution that covers *all* extension modules.

One approach could be to reserve the option prefix --enable-mod-XXX for this kind of configuration, allowing people to specify --disable-mod-zipimport (for example).

Another approach (closer to what we already have) would be to support a *disabled* marker in Modules/Setup (and Modules/Setup.local), so anybody wishing to disable modules could put

*disabled*
zipimport
_sre
unicodedata

into Modules/Setup.local (rather than specifying it on the configure command line).

[BreamoreBoy]

@thomas do you intend following this up as people certainly seem interested?

[koobs]

For what it's worth, we've been running with patch 0001 on FreeBSD's Python 3.4 port for 8 months with no issues [1].

It allows downstreams like us to easily customise what modules gets built, and offers substantial flexibility and benefits for various packaging scenarios.

This patch is the simplest approach that offers benefits without complicating the existing built infrastructure, or precluding a more 'elegant' solution if one is forthcoming in the future.

[1] https://svnweb.freebsd.org/ports/head/lang/python34/files/patch-issue20210

[tpetazzoni]

@mark I would be happy to, but if you refer to the previous discussion about this bug report, the feedback was quite negative. And since I'm not really willing to do some clean up to finally get the patches rejected, I'd like to at least have 1/ an agreement on the principle itself, and 2/ some guidance as to what needs to be changed in the currently proposed patches to get them accepted.

[krasimirvanev]

@thomas: Thank you for the patches! Adding more flexibility to the build system that allows for cross-compiling and building "embeddable/distributable" python is definitely something that needs to be done is my opinion. So I definitely find your work very valuable to me. I admit that I use all patches, related to feature configuration from http://git.buildroot.net/buildroot/tree/package/python
Thank you and keep up the great work! I'm positive about the direction that a flexibility/configurability should be in place, especially when talking for such a mature project.

[skrah]

Normally I hate environment variables, but perhaps in this case
PYTHON_DISABLE_MODULES="foo,bar,quux" would be sufficient?

Setup.py already has the "disabled_module_list" variable, it's
just a matter of setting this variable somehow.

This is really something for specialists, not general users.

[doko42]

fyi, this came up again at the 2016 Language Summit, the request here was to provide some minimal packages which are just enough to run a cloud image (the cloud-init package requires a somehow minimal python).

[tpetazzoni]

Hello,

On Sat, 04 Jun 2016 23:33:00 +0000, Matthias Klose wrote:

fyi, this came up again at the 2016 Language Summit, the request here
was to provide some minimal packages which are just enough to run a
cloud image (the cloud-init package requires a somehow minimal
python).

Thanks for the feedback! Definitely glad to hear that some other folks
are interested in reducing the footprint of the default Python
installation. Since embedded people have been ignored, hopefully the
cloud people will have more luck in this endeavor :)

Thanks!

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[vstinner]

Fedora 24 also uses a minimal "system python" to get small cloud
image: https://fedoraproject.org/wiki/Changes/System_Python#Benefit_to_Fedora

The base package only contains the minimum subset of the stdlib, the
full stdlib comes with a second package based on the first one.

[malemburg]

Just found this ticket and apparently late to the game...

Some comments:

It is already possible to selectively build Python C extension modules in the stdlib via editing the Modules/Setup file and this is a lot better to maintain than a long list of configure options.

Martin's suggestion to add support for a *disabled* option would make this even easier. This would also complement the *static* and *shared* option we have for Modules/Setup to define modules to compile into the main executable rather than as shared module.

The only bit that's missing (if really needed), is to prevent setup.py from building modules not listed in Modules/Setup as shared libs. This could be had with a single configure option, e.g. --disable-setuppy-builds.

But all that said, I don't think, it's really necessary to cripple Python just to save some image space. As you can see with our eGenix PyRun runtime, it is possible to compress Python down to just a few MBs:

http://www.egenix.com/products/python/PyRun/

while still having most of the stdlib readily available.

[vstinner]

Antoine Pitrou: "That's not really the point. The question is why we should have to maintain this ourselves. It is easy for interested people to maintain their own forks, especially when *removing* stuff."

It's painful to have to maintain downstream patches. Supporting this feature would make Python usable on more platforms.

Marc-Andre Lemburg: "it is possible to compress Python down to just a few MBs: (...)"

Right, that "a few MBs" can be too big on small devices, and it has a price on runtime performance, at least on the boot duration.

[pitrou]

It's painful to have to maintain downstream patches. Supporting this feature would make Python usable on more platforms.

Who volunteers to support it? It's a PITA to maintain the configure script and setup.py, and adding options only makes it more painful. This kind of niche features needs some dedicated maintainer so that other core developers don't have to bother with it.

[malemburg]

I'm still -1 on the approach taken by the OP, but +1 on Martin's approach of making such configurations more easily possible via Modules/Setup.

[vstinner]

Antoine Pitrou: " Who volunteers to support it? It's a PITA to maintain the configure script and setup.py, and adding options only makes it more painful. This kind of niche features needs some dedicated maintainer so that other core developers don't have to bother with it."

Oh, I forgot to put more context in my previous comment.

Xavier de Gaye opened the issue bpo-27640 "add the '--disable-test-suite' option to configure". He is a core developer working on porting CPython to Android. Xavier is the requested dedicated maintainer ;-)

[pitrou]

Victor, thanks for clarifying! That eliminates my objection, if there's a tacit agreement that the functionality can be broken by mistake and it's the dedicated maintainer (Xavier or someone else)'s job to repair it :-)

[doko42]

I don't see the point of having an option to do that. Will patches for third party projects be submitted as well, to not install the test suite? Or will pip gain such a feature? Note that removing the test suite completely, you'll make the test suites of some third party packages useless in some cases (I had a few bug reports for the Ubuntu packaging when I removed the tests unconditionally), but probably you don't care about these either.

[xdegaye]

The PR #132 implements the support of the *disabled* marker in a Setup file as suggested by Martin.

[Yhg1s]

This feature would also be nice to have for Python at Google: we want builds to be as reproducible as possible, avoid third-party libraries even when they are available on the build system, and avoid certain extension modules in any form. We currently delete them after building, but suppressing their builds makes much more sense.

[xdegaye]

New changeset c0364fc by xdegaye in branch 'master':
bpo-20210: Support the *disabled* marker in Setup files (GH-132)
c0364fc