How to run "msgfmt" (compiling po-files) while installing a project (pyproject.toml based) with "pip install ."?

[buhtz]

I have a Python project setup with pyproject.toml only. There are some po files in it.

For me it makes sense to not ship the mo-files (compiled po) with that project. The compilation should be done while installation. Technically it means that msgfmt need to be executed on each po file in the repo.

Can I achieve this somehow?

[jwodder]

No. By design, the installation procedure (the part that goes from a wheel to an installed package) does not support custom steps or execution of arbitrary code. Any compilation needs to be done either when building the sdist or when building the wheel.

[abravalheri]

The other way around. You can use setuptools to pre-generate .mo files that will go inside the .whl or the .tar.gz distribution archive. Note that the files are generated in the machine you "build" the package, not the machine of the final user.

You cannot use setuptools to chose where these files will be placed when the installation happens (they will always go to pre-defined places), or to generate them in the machine of the final user.

[buhtz]

Generating (means compiling po-files into mo-files) is my goal. How do I do this with setuptools? Where is the mechanic for this?

[abravalheri]

@buhtz, setuptools alone does not know how to do the conversion, but you can add a custom build step. Something like the following:

# setup.py
from setuptools immport Command, setup
from setuptools.command.build import build


class custom_build(build):
    sub_commands = [("my_build_step", None), *build.sub_commands]


class my_build_step(build):
    # See interface in https://setuptools.pypa.io/en/latest/userguide/extension.html

    def initialize_options(self):
        self.editable_mode = False
        self.build_lib = None
        ...

    def finalize_options(self):
        self.set_undefined_options('build', ('build_lib', 'build_lib'))
        ...

    def get_source_files(self): ...
    def get_outputs(self): ...
    def get_output_mapping(self): ...

    def run(self):
        # Here you can write your custom code and add files to f"{self.build_lib}/yourpackage"
        ...


setup(
   ...
   cmdclass={"build": custom_build, "my_build_step": my_build_step},
)

# Untested code, written from memory, please review :P

[buhtz]

That is a setup.py. I do use pyproject.toml only. There is no other way? Isn't the toml intended to be a replacement for setup.py/setup.cfg?

[abravalheri]

You can keep all the static parts in pyproject.toml and have a setup.py for the dynamic parts (running python setup.py ... directly as a CLI tool is deprecated, but setup.py as a configuration file is just fine that interoperates well with setup.cfg and pyproject.toml).

The dynamic behaviour needs to be programmed somewhere, if it is not in setup.py, then you need to find a setuptools plugin that will implement it for you¹ or write your own².

Footnotes

1. or write a wrapper, but if you are already writing custom code, I don't see why not use setup.py. At that point the setup.py approach is more straightforward and collectively the community has more experience dealing with it. ↩

2. Writing a plugin means that you have the customization code published as a PyPI package itself. Then the users can add it to the [build-system] requires list in pyproject.toml. An example of setuptools plugin for a different objective is setuptools-scm. ↩

[buhtz]

I have to warm the thread because I'm still not able to solve the problem.

I do have an example project (as a technical demo) to illustrate the case packaging a python project containing GNU gettext files. Please feel free to add Issues and PRs to it to improve it for other readers looking for help.

I wasn't able to get setuptools-gettext working. See Issue #26.
I also asked on python-list mailing list about the problem.

@abravalheri The custom build step looks interesting. But I don't know how to "execute" it or how modify the pyproject.toml.

Do I get that right if I can compile po to mo files in a custom build step I do "install" the mo files after it via something like this in the pyproject.toml file?

[options.package_data]
your_project = ["locale/*/LC_MESSAGES/*.mo"]

[abravalheri]

@buhtz, please have a look on #3912 (reply in thread) for how to configure custom build steps to be automatically executed as part of the build. Setuptools has long deprecated its "CLI tool" characteristics, so that is the only way currently supported for running custom build steps.

Do I get that right if I can compile po to mo files in a custom build step I do "install" the mo files after it via something like this in the pyproject.toml file?

If after the compilation, you have a file structure that is similar to (give or take a src directory associated with the src-layout):

<project-root OR `src`-root>
└── your_project/
    ├── locale/
    │   └── */
    │       ├── LC_MESSAGES
    │       └── *.mo
    └── other_files/
        └── *.py

and if the desired outcome is that the same file structure will be available under site-packages (i.e. replace <project-root OR src-root> with site-packages), then that seems right to me.

Do however keep in mind that your_project.*.LC_MESSAGES are considered by Python namespace packages, so you should add each one of them in the packages configuration.

[buhtz]

please have a look on #3912 (reply in thread) for how to configure

This did not help. Of course I tried that. But you write yourself that this was code out of your memory and untested. I don't understand your code. Lower case class names? class custom_build with a class attribute sub_commands only but pointing to class my_build_setup. But also the cmdclass= argument in setup() to point to my_build_setup.

I am at a point where code snippets do not help anymore.
I created an example project for that purpose. It would help a lot if you could provide a base solution (PR) based on that project.
https://codeberg.org/buhtz/tech-demo-python-packaging/src/branch/main/03_i18n_helloworld

[abravalheri]

Lower case class names?

Please let's not be judgmental here... You put whichever name you want in your classes.
If you look at the setuptools code (which has been existing for over a decade), this is the convention it uses, so I am just rolling with it.

class custom_build with a class attribute sub_commands only but pointing to class my_build_setup. But also the cmdclass= argument in setup() to point to my_build_setup.

Yes, and that does the magic. You need the cmdclass so that setuptools can map the string naming your command to the class implementation (since they might differ). You also need that to overwrite the default build with your custom class. Finally, you need the sub_commands to tell the "parent" command to automatically run the sub commands.

I am at a point where code snippets do not help anymore.

I am sorry, this is the best I have time to do can do for now. If you find out how to do this and would like to help future users facing the same problem, please feel free to open a PR adding more docs.

[buhtz]

from setuptools immport Command, setup  # <<< Why importing "Command". It seems to be unused?

# ...

class custom_build(build):
    # Do you "shadow" build.sub_commands with custom_build.sub_commands here?
    sub_commands = [
        ("my_build_step", None),  # <<< First appearance of "my_build_step" as a string
        *build.sub_commands
    ]


class my_build_step(build):
    # [snipped]

setup(
   ...
   cmdclass={
        "build": custom_build,   # There is no "run()" in custom_build. Why does it matter?
        "my_build_step": my_build_step  # <<< Second appearance of "my_build_step" as a string
    },
)

Why overwriting something? I want to add a custom build and not to replace the existing build procedure.
What is executed here in which order?

feel free to open a PR adding more docs.

That is what my example project is about. It documents on an example how to do it.

[abravalheri]

Why overwriting something? I want to add a custom build and not to replace the existing build procedure.

Indeed overwriting build is the way of adding a sub-command (i.e. the original build already has a fixed set of subcommands).
The overwriting happens at the configuration level for the cmdclass dict parameter. The implementation itself is not overwritten (in the example the original build class is left alone and a new subclass is created). The separation between the string with the command name and the implementation is done to allow that.

Do you "shadow" build.sub_commands with custom_build.sub_commands here?
"build": custom_build, # There is no "run()" in custom_build. Why does it matter?

This is object oriented programming and inheritance at play and the way Python handles class attributes. I will not go to specific details here because that is general Python programming, but I find that what works best for me to understand these things is to play around in the Python REPL and formulate/test hypothesis.

What is executed here in which order?

pip flow

1. You call pip install .
2. pip realises you are not passing the name of package for which it can download a .whl file, so it has to resort to building the .whl file locally.
3. pip asks setuptools to provide metadata (so it can find out, for example which dependencies are necessary)
4. pip asks setuptools to create an sdist from the local directory => see setuptools flow below
5. pip asks setuptools to create a .whl file from the sdist (i.e. .tar.gz file) => see setuptools flow below
6. pip gets the .whl file that was generated by setuptools and unzips it. Then pip move the directories and files to predefined specific locations at the end user's machine. You don't have fine control over these specific locations.

setuptools flow

1. Run setup.py to create a Distribution object (if you don't have one a default is used).
2. Read available configuration files
3. Run a command (one of egg_info, dist_info, sdist, bdist_wheel, depending on how pip calls setuptools).

bdist_wheel flow (implemented in the pypa/wheel repository, so I don't know 100% of the details, please feel free to read the bdist_wheel.py code in that repository to learn more details)

1. ... a bunch of stuff ...
2. Run the build command (either the default one provided by setuptools, or the one you are overwriting)
   • Here build will run each one of the commands in the sub_command list in sequence.
3. ... a bunch of stuff ...
4. Collects files and add them to a .whl archive (which is a zip).
   • The directory structure in this file reflects more or less the directory structure in your project.