Recommended way to add custom build steps #3762

jarkenau · 2023-01-10T18:38:05Z

jarkenau
Jan 10, 2023

My current setup requires adding some custom build steps, e.g., for generating code from .proto files. My current approach is to add commands to the setup.py class and configure all the other settings in the pyproject.toml :

class BuildProtobuf(setuptools.Command):
    """Custom build command."""

    user_options = [ ]

    def initialize_options(self) -> None:
        ...

    def finalize_options(self) -> None:
        ...

    def run(self) -> None:
        ...
...
 setuptools.setup(
    packages=[...],
    cmdclass={
        'build_proto': BuildProtobuf,
        ...
    },
)

And then run the commands with python3 setup.py build_proto sdist. Now, since the direct usage of setup.py is considered deprecated I'm not sure how to approach this differently. Are there currently any other preferred ways to handle this?

#3422 looks connected, but I couldn't understand the approach and maybe there are some updates since.

Answered by abravalheri

Jan 11, 2023

Hi @jarkenau,

I can successfully run the following:

rm -rf /tmp/myproj
mkdir /tmp/myproj
cd /tmp/myproj
cat <<EOF > setup.py
from contextlib import suppress
from pathlib import Path
from setuptools import Command, setup
from setuptools.command.build import build

class CustomCommand(Command):
    def initialize_options(self) -> None:
        self.bdist_dir = None

    def finalize_options(self) -> None:
        with suppress(Exception):
            self.bdist_dir = Path(self.get_finalized_command("bdist_wheel").bdist_dir)

    def run(self) -> None:
        if self.bdist_dir:
            self.bdist_dir.mkdir(parents=True, exist_ok=True)
            (self.bdist_dir / "file.txt").write_text…

View full answer

webknjaz · 2023-01-10T18:45:49Z

webknjaz
Jan 10, 2023
Collaborator

You can make an in-tree build backend that would be a thin wrapper around the setuptools' one. You can also make use of PEP 517 config_settings there.

I've made something similar in the ansible/pylibssh repo and I'm working on another such thing in ansible/ansible now.

10 replies

webknjaz Jan 10, 2023
Collaborator

Although, unless you have a case for building protobuf only in some situations, maybe just do that unconditionally in the build_wheel hook?

jarkenau Jan 10, 2023
Author

Ok, I think I got the general idea now.

Although, unless you have a case for building protobuf only in some situations, maybe just do that unconditionally in the build_wheel hook?

Yes I cloud try that.

Also, to raise another point, this process seems like a lot of work for people who are not familiar with setuptools and other python build stuff. For that reason, my first thought was to have a regular python script to prepare the packages as needed, then call the build scripts. What's your opinion on this?

webknjaz Jan 10, 2023
Collaborator

Don't do that. These interfaces are standardized in PEP 517 and are invoked by pip and pypa/build, and any other PEP 517 build front-end. Forcing people to run an additional script places them in a position where they won't be able to simply invoke pip install https://github.com/your/prj.git or add it into their deps, they'll be forced to clone the project manually, build it manually using unconventional methods with a lot of mental overhead involving going through the build docs, and only then being able to use it. Essentially, you'd be forcing end-users to learn the internals of non-standard packaging they'll never need.

jarkenau Jan 12, 2023
Author

Definitly makes sense for a full python project. In my case we just provide the generated python gRPC API for the ease of use. No one would install the hole project with pip install https://github.com/your/prj.git or extend the python classes in any way. Special thanks for the fast replies 👍

webknjaz Jan 13, 2023
Collaborator

Even if you just publish it to PyPI, it makes sense. The downstream distributions may need to rebuild from scratch, for example — adding anything that works outside the standard and is not integrated into the PEP 517 process would ultimately increase maintenance burden for someone.

abravalheri · 2023-01-11T16:13:52Z

abravalheri
Jan 11, 2023
Maintainer

Hi @jarkenau,

I can successfully run the following:

rm -rf /tmp/myproj
mkdir /tmp/myproj
cd /tmp/myproj
cat <<EOF > setup.py
from contextlib import suppress
from pathlib import Path
from setuptools import Command, setup
from setuptools.command.build import build

class CustomCommand(Command):
    def initialize_options(self) -> None:
        self.bdist_dir = None

    def finalize_options(self) -> None:
        with suppress(Exception):
            self.bdist_dir = Path(self.get_finalized_command("bdist_wheel").bdist_dir)

    def run(self) -> None:
        if self.bdist_dir:
            self.bdist_dir.mkdir(parents=True, exist_ok=True)
            (self.bdist_dir / "file.txt").write_text("hello world", encoding="utf-8")

class CustomBuild(build):
    sub_commands = [('build_custom', None)] + build.sub_commands

setup(cmdclass={'build': CustomBuild, 'build_custom': CustomCommand})
EOF

cat <<EOF > pyproject.toml
[build-system]
requires = ['setuptools']
build-backend = 'setuptools.build_meta'

[project]
name = 'hello-world'
version = '0.1'
EOF

python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip build
.venv/bin/python -m build
unzip -l dist/*.whl
# Archive:  dist/hello_world-0.1-py3-none-any.whl
#   Length      Date    Time    Name
# ---------  ---------- -----   ----
#        11  2023-01-11 16:07   file.txt   # <---------
#        54  2023-01-11 16:07   hello_world-0.1.dist-info/METADATA
#        92  2023-01-11 16:07   hello_world-0.1.dist-info/WHEEL
#         1  2023-01-11 16:07   hello_world-0.1.dist-info/top_level.txt
#       366  2023-01-11 16:07   hello_world-0.1.dist-info/RECORD
# ---------                     -------
#       524                     5 files

Have you considered a similar approach?

4 replies

jarkenau Jan 12, 2023
Author

This should work for most people, that's why I will mark it as the answer.

jarkenau Jan 12, 2023
Author

For me, it's not working since the python packages are generated during the build, so I don't have any packages to put here:

setup(
  packages=[?],
  cmdclass={'build': CustomBuild, 'build_custom': CustomCommand},
)

def run(self) -> None:
  def run(self) -> None:
  proto_files = []
  if Path(self.proto_msgs_path).is_dir():
      for path in Path(self.proto_msgs_path).glob('*.proto'):
          proto_files.append(str(path))
  
  # create package structure
  ....
  
  # generate python classes
  protoc_call = [
  "python3",
  "-m",
  "grpc_tools.protoc",
  f"--proto_path={self.proto_msgs_path}",
  f"--python_out={self.output_dir}",
  *proto_files,
  ]
  subprocess.call(protoc_call)

It looks like I'm trying to achieve something, setuptools is not really made for. I think I will go for a second script which builds all the python files and live with the downsides pointed out by @webknjaz above

abravalheri Jan 12, 2023
Maintainer

Hi @jarkenau do you need to put anything there?

For example, what is the conceptual problem with the following approach?

rm -rf /tmp/myproj
mkdir -p /tmp/myproj/proto
cd /tmp/myproj
cat <<EOF > setup.py
import subprocess
from contextlib import suppress
from pathlib import Path
from setuptools import Command, setup
from setuptools.command.build import build

class CustomCommand(Command):
    def initialize_options(self) -> None:
        self.bdist_dir = None
        self.proto_msgs_path = None
        self.pkg_name = None

    def finalize_options(self) -> None:
        self.pkg_name = self.distribution.get_name().replace('-', '_')
        self.proto_msgs_path = Path("proto")
        with suppress(Exception):
            self.bdist_dir = Path(self.get_finalized_command("bdist_wheel").bdist_dir)

    def get_source_files(self) ->'list[str]':
        if self.proto_msgs_path.is_dir():
            return [str(path) for path in self.proto_msgs_path.glob('*.proto')]
        else:
            return []

    def run(self) -> None:
        if self.bdist_dir:
            # Create package structure
            output_dir = self.bdist_dir / self.pkg_name
            output_dir.mkdir(parents=True, exist_ok=True)
            # generate python classes
            protoc_call = [
                "python3",
                "-m",
                "grpc_tools.protoc",
                f"--proto_path={self.proto_msgs_path}",
                f"--python_out={output_dir}",
                *self.get_source_files(),
            ]
            subprocess.call(protoc_call)

class CustomBuild(build):
    sub_commands = [('build_custom', None)] + build.sub_commands

setup(
   packages=[],
   cmdclass={'build': CustomBuild, 'build_custom': CustomCommand}
)
EOF

cat <<EOF > pyproject.toml
[build-system]
requires = ['setuptools', 'grpcio-tools>=1.51.1']
build-backend = 'setuptools.build_meta'

[project]
name = 'hello-world'
version = '0.1'
EOF

cat <<EOF > proto/person.proto
message Person {
  optional string name = 1;
  optional int32 id = 2;
  optional string email = 3;
}
EOF

python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip build
.venv/bin/python -m build
# Successfully built hello-world-0.1.tar.gz and hello_world-0.1-py3-none-any.whl
unzip -l dist/*.whl
# Archive:  dist/hello_world-0.1-py3-none-any.whl
#   Length      Date    Time    Name
# ---------  ---------- -----   ----
#       975  2023-01-12 14:13   hello_world/person_pb2.py
#        54  2023-01-12 14:13   hello_world-0.1.dist-info/METADATA
#        92  2023-01-12 14:13   hello_world-0.1.dist-info/WHEEL
#         1  2023-01-12 14:13   hello_world-0.1.dist-info/top_level.txt
#       384  2023-01-12 14:13   hello_world-0.1.dist-info/RECORD
# ---------                     -------
#      1506                     5 files
tar tvf dist/*.tar.gz
# drwxr-xr-x abravalheri/abravalheri 0 2023-01-12 14:12 hello-world-0.1/
# -rw-r--r-- abravalheri/abravalheri 53 2023-01-12 14:12 hello-world-0.1/PKG-INFO
# drwxr-xr-x abravalheri/abravalheri  0 2023-01-12 14:12 hello-world-0.1/hello_world.egg-info/
# -rw-r--r-- abravalheri/abravalheri 53 2023-01-12 14:12 hello-world-0.1/hello_world.egg-info/PKG-INFO
# -rw-r--r-- abravalheri/abravalheri 182 2023-01-12 14:12 hello-world-0.1/hello_world.egg-info/SOURCES.txt
# -rw-r--r-- abravalheri/abravalheri   1 2023-01-12 14:12 hello-world-0.1/hello_world.egg-info/dependency_links.txt
# -rw-r--r-- abravalheri/abravalheri   1 2023-01-12 14:12 hello-world-0.1/hello_world.egg-info/top_level.txt
# drwxr-xr-x abravalheri/abravalheri   0 2023-01-12 14:12 hello-world-0.1/proto/
# -rw-r--r-- abravalheri/abravalheri 101 2023-01-12 14:11 hello-world-0.1/proto/person.proto
# -rw-r--r-- abravalheri/abravalheri 153 2023-01-12 14:11 hello-world-0.1/pyproject.toml
# -rw-r--r-- abravalheri/abravalheri  38 2023-01-12 14:12 hello-world-0.1/setup.cfg
# -rw-r--r-- abravalheri/abravalheri 1771 2023-01-12 14:11 hello-world-0.1/setup.py

Also note that for Python, any folder is a valid import package as long as its name is a valid Python identifier, even if it does not contain Python files, so the following would also work:

rm -rf /tmp/myproj
mkdir -p /tmp/myproj/hello_world
cd /tmp/myproj
cat <<EOF > setup.py
import subprocess
from contextlib import suppress
from pathlib import Path
from setuptools import Command, setup
from setuptools.command.build import build

class CustomCommand(Command):
    def initialize_options(self) -> None:
        self.bdist_dir = None
        self.proto_msgs_path = None
        self.pkg_name = None

    def finalize_options(self) -> None:
        self.pkg_name = self.distribution.get_name().replace('-', '_')
        self.proto_msgs_path = Path(self.pkg_name)
        with suppress(Exception):
            self.bdist_dir = Path(self.get_finalized_command("bdist_wheel").bdist_dir)

    def get_source_files(self) ->'list[str]':
        if self.proto_msgs_path.is_dir():
            return [str(path) for path in self.proto_msgs_path.glob('*.proto')]
        else:
            return []

    def run(self) -> None:
        if self.bdist_dir:
            # Create package structure
            output_dir = self.bdist_dir / self.pkg_name
            output_dir.mkdir(parents=True, exist_ok=True)
            # generate python classes
            protoc_call = [
                "python3",
                "-m",
                "grpc_tools.protoc",
                f"--proto_path={self.proto_msgs_path}",
                f"--python_out={output_dir}",
                *self.get_source_files(),
            ]
            subprocess.call(protoc_call)

class CustomBuild(build):
    sub_commands = [('build_custom', None)] + build.sub_commands

setup(
   cmdclass={'build': CustomBuild, 'build_custom': CustomCommand}
)
EOF

cat <<EOF > pyproject.toml
[build-system]
requires = ['setuptools', 'grpcio-tools>=1.51.1']
build-backend = 'setuptools.build_meta'

[project]
name = 'hello-world'
version = '0.1'

[tool.setuptools]
packages = ['hello_world']
exclude-package-data = {hello_world = ["*.proto"]}  # don't include proto files in wheel
EOF

cat <<EOF > hello_world/person.proto
message Person {
  optional string name = 1;
  optional int32 id = 2;
  optional string email = 3;
}
EOF

python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip build
.venv/bin/python -m build
unzip -l dist/*.whl
# Archive:  dist/hello_world-0.1-py3-none-any.whl
#   Length      Date    Time    Name
# ---------  ---------- -----   ----
#       975  2023-01-12 15:22   hello_world/person_pb2.py
#        54  2023-01-12 15:22   hello_world-0.1.dist-info/METADATA
#        92  2023-01-12 15:22   hello_world-0.1.dist-info/WHEEL
#        12  2023-01-12 15:22   hello_world-0.1.dist-info/top_level.txt
#       385  2023-01-12 15:22   hello_world-0.1.dist-info/RECORD
# ---------                     -------
#      1518                     5 files
tar tvf dist/*.tar.gz
# drwxr-xr-x abravalheri/abravalheri 0 2023-01-12 15:21 hello-world-0.1/
# -rw-r--r-- abravalheri/abravalheri 53 2023-01-12 15:21 hello-world-0.1/PKG-INFO
# drwxr-xr-x abravalheri/abravalheri  0 2023-01-12 15:21 hello-world-0.1/hello_world/
# -rw-r--r-- abravalheri/abravalheri 101 2023-01-12 15:20 hello-world-0.1/hello_world/person.proto
# drwxr-xr-x abravalheri/abravalheri   0 2023-01-12 15:21 hello-world-0.1/hello_world.egg-info/
# -rw-r--r-- abravalheri/abravalheri  53 2023-01-12 15:21 hello-world-0.1/hello_world.egg-info/PKG-INFO
# -rw-r--r-- abravalheri/abravalheri 188 2023-01-12 15:21 hello-world-0.1/hello_world.egg-info/SOURCES.txt
# -rw-r--r-- abravalheri/abravalheri   1 2023-01-12 15:21 hello-world-0.1/hello_world.egg-info/dependency_links.txt
# -rw-r--r-- abravalheri/abravalheri  12 2023-01-12 15:21 hello-world-0.1/hello_world.egg-info/top_level.txt
# -rw-r--r-- abravalheri/abravalheri 288 2023-01-12 15:20 hello-world-0.1/pyproject.toml
# -rw-r--r-- abravalheri/abravalheri  38 2023-01-12 15:21 hello-world-0.1/setup.cfg
# -rw-r--r-- abravalheri/abravalheri 1540 2023-01-12 15:20 hello-world-0.1/setup.py#

If you like, you can even add a hello_world/__init__.py...

I am trying to understand why do you think this is not achievable by setuptools...

jarkenau Jan 13, 2023
Author

I got some error, that setuptools can't decide between packages or something. I can't remember or reproduce it, but this just works out off the box, thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recommended way to add custom build steps #3762

{{title}}

Replies: 2 comments 14 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Recommended way to add custom build steps #3762

jarkenau Jan 10, 2023

Replies: 2 comments · 14 replies

webknjaz Jan 10, 2023 Collaborator

webknjaz Jan 10, 2023 Collaborator

jarkenau Jan 10, 2023 Author

webknjaz Jan 10, 2023 Collaborator

jarkenau Jan 12, 2023 Author

webknjaz Jan 13, 2023 Collaborator

abravalheri Jan 11, 2023 Maintainer

jarkenau Jan 12, 2023 Author

jarkenau Jan 12, 2023 Author

abravalheri Jan 12, 2023 Maintainer

jarkenau Jan 13, 2023 Author

jarkenau
Jan 10, 2023

Replies: 2 comments 14 replies

webknjaz
Jan 10, 2023
Collaborator

webknjaz Jan 10, 2023
Collaborator

jarkenau Jan 10, 2023
Author

webknjaz Jan 10, 2023
Collaborator

jarkenau Jan 12, 2023
Author

webknjaz Jan 13, 2023
Collaborator

abravalheri
Jan 11, 2023
Maintainer

jarkenau Jan 12, 2023
Author

jarkenau Jan 12, 2023
Author

abravalheri Jan 12, 2023
Maintainer

jarkenau Jan 13, 2023
Author