Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for custom build steps #2591

Open
jaraco opened this issue Mar 5, 2021 · 6 comments
Open

Support for custom build steps #2591

jaraco opened this issue Mar 5, 2021 · 6 comments

Comments

@jaraco
Copy link
Member

jaraco commented Mar 5, 2021

I've seen cases where users wish to add custom steps to the build process (example, example). Historically, users have managed to work around the issue by overriding the build or install commands (using cmdclass), but this approach requires replacing the default command just to inject some behavior. Preferable would be to avoid having to override the default behavior and for Setuptools to present a protocol for extending build steps.

And indeed, distutils already provides a mechanism to extend build steps through sub_commands.

My proposal is that the build command present an interface to append or prepend commands to the build subcommands such that a third-party provider could provide an independent command, expose it through distutils.commands, and either implicitly as part of the import or explicitly by the end user, append or prepend that command to the subcommands.

I think it should be possible to demonstrate this behavior today just by mutating the sub_commands list directly, though I'd probably recommend a proper API for users to use.

@jaraco
Copy link
Member Author

jaraco commented Mar 5, 2021

In demo-custom, I've created a project that relies on a custom-build command without any overriding of build or install commands.

This approach required the explicit addition of the command in the setup.py for demo-custom.

But it does demo the concept:

draft $ pip wheel -v git+https://github.com/jaraco/demo-custom 2>&1 | tail -n 32
Created temporary directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-unpack-4tfm8tmb
Building wheels for collected packages: demo-custom
  Created temporary directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-xpii40jz
  Destination directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-xpii40jz
  Building wheel for demo-custom (PEP 517): started
  Running command /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmp3izxy0r8
  running bdist_wheel
  running build
  running custom-build
  brrrrrrrrr
  installing to build/bdist.macosx-10.9-x86_64/wheel
  running install
  running install_egg_info
  running egg_info
  writing demo_custom.egg-info/PKG-INFO
  writing dependency_links to demo_custom.egg-info/dependency_links.txt
  writing top-level names to demo_custom.egg-info/top_level.txt
  writing manifest file 'demo_custom.egg-info/SOURCES.txt'
  Copying demo_custom.egg-info to build/bdist.macosx-10.9-x86_64/wheel/demo_custom-0.1.dev1+gffa9928-py3.9.egg-info
  running install_scripts
  creating build/bdist.macosx-10.9-x86_64/wheel/demo_custom-0.1.dev1+gffa9928.dist-info/WHEEL
  creating '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-wheel-xpii40jz/tmpi7uk802o/demo_custom-0.1.dev1+gffa9928-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-x86_64/wheel' to it
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/METADATA'
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/WHEEL'
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/top_level.txt'
  adding 'demo_custom-0.1.dev1+gffa9928.dist-info/RECORD'
  removing build/bdist.macosx-10.9-x86_64/wheel
  Building wheel for demo-custom (PEP 517): finished with status 'done'
  Created wheel for demo-custom: filename=demo_custom-0.1.dev1+gffa9928-py3-none-any.whl size=1147 sha256=e9ce3ed5131a843264242225b5e80c25e4a4b1e39f3eddedb5d946208951eafa
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-o0hxe9tb/wheels/2b/94/c3/ca414870dfa5f5a106895a32bc0b3837efe8cda48776c7cf71
Successfully built demo-custom
Removed build tracker: '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-tracker-2ku320jw'

Note running custom-build and brrrrrrrrr.

@Nick-Hall
Copy link

This functionality would be useful for building the Gramps software. We have additional build steps for translations and manual pages.

@basil96
Copy link

basil96 commented Sep 24, 2021

Thank you, @jaraco , for providing an example of this functionality. I was struggling to implement a custom build step for PyInstaller using cmdclass in setup.cfg with a custom override of build. I tried implementing that using #2571, but it fails for editable pip install -e . installs with a ModuleNotFoundError when it encounters the name of the custom module in setup.cfg.

Your approach as shown here is much cleaner and actually works. Looking forward to the finalized API.

abravalheri added a commit to abravalheri/setuptools that referenced this issue Nov 20, 2021
As described in pypa#2591, users might wish to add custom steps to the build
process, and this can be done via sub-commands.

The changes implemented here expose a new set of entry-points
to facilitate this process. This allows the implementation of custom
steps in 3rd-party plugins.
abravalheri added a commit to abravalheri/bokeh that referenced this issue Jun 9, 2022
abravalheri added a commit to abravalheri/setuptools that referenced this issue Jun 24, 2022
As described in pypa#2591, users might wish to add custom steps to the build
process, and this can be done via sub-commands.

The changes implemented here expose a new set of entry-points
to facilitate this process. This allows the implementation of custom
steps in 3rd-party plugins.
@abravalheri
Copy link
Contributor

This approach required the explicit addition of the command in the setup.py for demo-custom.

I think we can get around requiring the users to explicitly add the sub-command, by changing the plugin to use finalize_distribution_option to automatically insert the sub-command.

I did created a proof of concept in https://github.com/abravalheri/experiment-setuptools-plugin.

@jaraco what do you think of this approach?

I think in the long term we will want to go for a different entry-point, however, this solution seems to be usable nowadays without any extra change.

abravalheri added a commit to abravalheri/setuptools-experiment-custom-build that referenced this issue Jun 24, 2022
The comment in pypa/setuptools#2591 (comment)
reports that users wanting to use the custom-command plugin need to
explicitly add it using `setup.py`.

This commit tried to demonstrate how that can be done without requiring
the user to change their `setup.py`.

The "installation" of the new sub-command is done via the
`finalize_distribution_options` hook.
abravalheri added a commit to abravalheri/setuptools that referenced this issue Jun 26, 2022
As described in pypa#2591, users might wish to add custom steps to the build
process, and this can be done via sub-commands.

The changes implemented here expose a new set of entry-points
to facilitate this process. This allows the implementation of custom
steps in 3rd-party plugins.
abravalheri added a commit to abravalheri/setuptools that referenced this issue Feb 27, 2023
As described in pypa#2591, users might wish to add custom steps to the build
process, and this can be done via sub-commands.

The changes implemented here expose a new set of entry-points
to facilitate this process. This allows the implementation of custom
steps in 3rd-party plugins.
@ax3l
Copy link

ax3l commented Aug 17, 2023

With Python 3.12 on our doorstep, I am searching right now for recommended ways to replace (#2928)
https://github.com/AMReX-Codes/pyamrex/blob/23.08/setup.py#L8-L9

from distutils.command.build import build
from distutils.command.clean import clean

for steps such as
https://github.com/AMReX-Codes/pyamrex/blob/23.08/setup.py#L22C1-L47C1

class CopyPreBuild(build):
    def initialize_options(self):
        build.initialize_options(self)
        # We just overwrite this because the default "build" (and "build/lib")
        # clashes with directories many developers have in their source trees;
        # this can create confusing results with "pip install .", which clones
        # the whole source tree by default
        self.build_base = "_tmppythonbuild"

    def run(self):
        # remove existing build directory
        c = clean(self.distribution)
        # ...

I do not understand the example above to replace these with setuptools, as the migration guide suggests
https://peps.python.org/pep-0632/#migration-advice

Since #2928 is closed, what is the current recommended practice to migrate such logic? :)

@abravalheri
Copy link
Contributor

abravalheri commented Aug 17, 2023

Hi @ax3l , if you are using the latest versions of setuptools the existing code should still keep working even on Python 3.12.

You can use setuptools.command.build.build and maybe shutil.rmtree to replace the distutils imports if you want? Does that work for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants