Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --disable-test-modules configure option to not build nor install tests #71827

Closed
xdegaye mannequin opened this issue Jul 28, 2016 · 14 comments
Closed

Add --disable-test-modules configure option to not build nor install tests #71827

xdegaye mannequin opened this issue Jul 28, 2016 · 14 comments
Labels
3.10 build type-feature

Comments

@xdegaye
Copy link
Mannequin

@xdegaye xdegaye mannequin commented Jul 28, 2016

BPO 27640
Nosy @doko42, @vstinner, @vadmium, @tpetazzoni, @edmorley, @moreati, @eli-schwartz, @pxinwr
PRs
  • #23886
  • Files
  • disable-test-suite.patch
  • listdirs.py
  • pr23886_files.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-12-30.12:56:02.776>
    created_at = <Date 2016-07-28.10:05:17.918>
    labels = ['type-feature', '3.10', 'build']
    title = 'Add --disable-test-modules configure option to not build nor install tests'
    updated_at = <Date 2021-02-21.11:09:37.921>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2021-02-21.11:09:37.921>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-12-30.12:56:02.776>
    closer = 'vstinner'
    components = ['Build', 'Cross-Build']
    creation = <Date 2016-07-28.10:05:17.918>
    creator = 'xdegaye'
    dependencies = []
    files = ['43917', '43918', '49707']
    hgrepos = []
    issue_num = 27640
    keywords = ['patch']
    message_count = 14.0
    messages = ['271514', '271526', '271619', '287515', '287525', '304721', '336937', '384034', '384063', '384064', '386758', '386823', '387439', '387454']
    nosy_count = 8.0
    nosy_names = ['doko', 'vstinner', 'martin.panter', 'thomas-petazzoni', 'edmorley', 'Alex.Willmer', 'eschwartz', 'pxinwr']
    pr_nums = ['23886']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue27640'
    versions = ['Python 3.10']

    @xdegaye
    Copy link
    Mannequin Author

    @xdegaye xdegaye mannequin commented Jul 28, 2016

    Do not install the Python test suite when configure is run with '--disable-test-suite'. This about divides by two the size of the installed standard library.

    Related to bpo-26852: add the '--enable-legacy-pyc-files' option to configure.

    The attached listdirs.py script may be used to check that the set of sub-directories in the std library is not changed by the patch.

    @xdegaye xdegaye mannequin self-assigned this Jul 28, 2016
    @xdegaye xdegaye mannequin added build type-feature labels Jul 28, 2016
    @tpetazzoni
    Copy link
    Mannequin

    @tpetazzoni tpetazzoni mannequin commented Jul 28, 2016

    We have a similar patch in Buildroot (see https://git.buildroot.org/buildroot/tree/package/python3/0017-Add-an-option-to-disable-installation-of-test-module.patch) so we would be very happy to see this patch merged.

    Note that we also have many more patches to disable various parts of the Python standard installation (https://git.buildroot.org/buildroot/tree/package/python3/). So either we have one option for each feature (like we have implemented), or a more general config option --disable-feature=test,this,that.

    See also bug http://bugs.python.org/issue20210 which is related, and was also proposing a patch to disable the test suite.

    @xdegaye
    Copy link
    Mannequin Author

    @xdegaye xdegaye mannequin commented Jul 29, 2016

    Thanks for the review Martin. I will update the patch accordingly later.

    @xdegaye xdegaye mannequin removed their assignment Feb 3, 2017
    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Feb 10, 2017

    I'm in favor of allowing to build a "subset" of Python. Not only it makes sense for embedded devices, but in many cases it's even not possible to do otherwise: "full" Python is too big for small devices.

    But I would prefer to see a public discussion on python-dev before going into this direction.

    The previous attempt failed because some core developers were opposed to support officially a "subset of Python". But since this previous discussion (two years ago? longer?), MicroPython became popular, and this implementation of Python has a small subset of the stdlib (but it supports asyncio ;-)) Maybe it's time for a new attempt :-) Oh, and the meanwhile we also made progress on the Android port, which is now an important platform.

    In short, I would like to official support Python subsets ;-) At least, add options for that.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Feb 10, 2017

    But I would prefer to see a public discussion on python-dev before going into this direction.

    Ah, the discussion restarted on the other issue:
    http://bugs.python.org/issue20210#msg287516

    @xdegaye
    Copy link
    Mannequin Author

    @xdegaye xdegaye mannequin commented Oct 21, 2017

    Ah, the discussion restarted on the other issue: http://bugs.python.org/issue20210#msg287516

    bpo-20210 has ended up as a change in the build system eventually and not the installation process, only a tiny step towards the configuration of a smaller Python distribution :(

    After browsing again the last discussion on "[Python-ideas] size of the installation of Python on mobile devices" [1], I find all the arguments in favor of such changes still relevant. I would also add the following new arguments:

    • One frequent argument used against those changes is that "[packagers for mobile and embedded devices] will have to patch the code base anyway to get things working in such an environment". But this is not true, see bpo-30386 (Add a build infrastructure for Android) that proposes an addition to Python that does not modify a SINGLE LINE in the existing source code. So it is not true (or at least not anymore true) that packagers for mobile and embedded devices need to patch the code base anyway.

    • When Python is installed by the termux [2] Android application on an Android device, the size of the installed standard library is 34M. The termux installation script does not install the following files [3]:

      TERMUX_PKG_RM_AFTER_INSTALL="
      bin/python${_MAJOR_VERSION}m bin/idle*
      lib/python${_MAJOR_VERSION}/idlelib
      lib/python${_MAJOR_VERSION}/test
      lib/python${_MAJOR_VERSION}/tkinter
      lib/python${_MAJOR_VERSION}/turtledemo
      "

    But it fails to not install also:
    lib/python${_MAJOR_VERSION}/config-${_MAJOR_VERSION}m
    lib/python${_MAJOR_VERSION}/ctypes/test
    lib/python${_MAJOR_VERSION}/distutils/tests
    lib/python${_MAJOR_VERSION}/sqlite3/test
    lib/python${_MAJOR_VERSION}/unittest/test

    This is not consistent and should be fixed by a proper (endorsed by Python) installation process.

    [1] https://mail.python.org/pipermail/python-ideas/2016-July/
    [2] https://termux.com/
    [3] https://github.com/termux/termux-packages/blob/master/packages/python/build.sh

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Mar 1, 2019

    I modified setup.py to add a new TEST_EXTENSIONS which allows to not build test extensions:

    New changeset cfe172d by Victor Stinner in branch 'master':
    bpo-36146: Add TEST_EXTENSIONS to setup.py (GH-12129)
    cfe172d

    setup.py should be modified manually, but it's a small step towards be able to build Python without tests ;-)

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Dec 29, 2020

    pr23886_files.diff: see #23886 (comment)

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Dec 30, 2020

    New changeset 277ce30 by pxinwr in branch 'master':
    bpo-27640: Add --disable-test-modules configure option (GH-23886)
    277ce30

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Dec 30, 2020

    Thanks all, it's now fixed in the master branch, the future Python 3.10.

    Thanks Xavier de Gaye for the initial patch and reported the issue.

    Thanks Peixing Xin for the final change which is now even better!

    Thanks also Thomas Petazzoni who maintained the downstream patch in buildroot.

    See also discussion on PR 23886 about the technical change and the different tests that we run to ensure that the change doesn't break anything, and that it works as expected.

    I'm really not comfortable at all to change the Python build system. It's very complex and fragile, I don't know it very well, so I'm not confident in what I am doing. For these reasons, I'm strongly against backporting the change to stable branches (3.8 and 3.9) and so I close the issue

    @vstinner vstinner added the 3.10 label Dec 30, 2020
    @vstinner vstinner changed the title add the '--disable-test-suite' option to configure Add --disable-test-modules configure option to not build nor install tests Dec 30, 2020
    @eli-schwartz
    Copy link
    Mannequin

    @eli-schwartz eli-schwartz mannequin commented Feb 10, 2021

    I have a slightly different interest in declining to install tests... Arch Linux provides the tests as a split subpackage with currently some hacky shell globbing to delete the tests after make install, then do a second libinstall and delete everything but the tests.

    It would be more convenient if we had a Makefile option to install the tests separately.

    It's now possible to disable installing tests with a filthy hack: make install TEST_MODULES= TESTSUBDIRS=

    However, it's not possible to similarly hack make libinstall LIBSUBDIRS=
    Because it installs some toplevel stuff, then runs lib2to3.pgen2.driver on the grammar that got skipped by the LIBSUBDIRS= hack.

    Thoughts on maybe splitting this up at install time? Would probably mean duplicating some of libinstall as testinstall...

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Feb 11, 2021

    It would be more convenient if we had a Makefile option to install the tests separately.

    While this use case is tighly coupled to this feature, would you mind to open a separate issue for your use case? IMO it makes sense and it sounds easy to implement it (especially after --disable-test-modules feature has been implemented).

    @eli-schwartz
    Copy link
    Mannequin

    @eli-schwartz eli-schwartz mannequin commented Feb 21, 2021

    I filed bpo-43282 for this.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Feb 21, 2021

    I filed bpo-43282 for this.

    Thanks ;-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 build type-feature
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant