Feature request: hatchling should implement the prepare_metadata_for_build_wheel hook

[musicinmybrain]

In Fedora Linux, the prepare_metadata_for_build_wheel hook is used to generate a package’s runtime requirements to use as build-time RPM dependencies, which is helpful for ensuring everything is in place to run the tests. Since hatchling does not implement this hook, it’s not possible to do this for hatchling or for projects that use it as a build backend. The most popular Python build systems (setuptools, poetry, flit, build, etc.) do support this hook.

This is just an inconvenience, and can be worked around by some manual effort or project-specific scripting hacks, but it would be a helpful feature if it’s practical to implement.

[ofek]

I can do this. Is the METADATA file all you need?

[ofek]

Per https://www.python.org/dev/peps/pep-0517/#build-wheel:

If the build frontend has previously called prepare_metadata_for_build_wheel and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the created .dist-info directory as the metadata_directory argument. If this argument is provided, then build_wheel MUST produce a wheel with identical metadata.

So, without access to build-time data METADATA, entry_points.txt, and license_files/ would be identical but WHEEL would not since build hooks can modify the tag/file name.

Therefore, we have three options:

1. Go against PEP 517 and produce a non-identical .dist-info directory

2. Go against PEP 427 and omit the WHEEL file

3. Per https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel:

   The hook MAY also create other files inside this directory, and a build frontend MUST preserve, but otherwise ignore, such files

   We could actually build and cache the wheel here, and extract the real .dist-info directory. The build_wheel hook would then simply reuse the wheel from this stage.

[FFY00]

Since hatchling does not implement this hook, it’s not possible to do this for hatchling or for projects that use it as a build backend.

You can just build the wheel in that case, the PEP even says that. We even provide a helper that does just that in pypa/build, https://pypa-build.readthedocs.io/en/latest/api.html#build.util.project_wheel_metadata.

3. We could actually build and cache the wheel here, and extract the real .dist-info directory. The build_wheel hook would then simply reuse the wheel from this stage.

This does not make sense, as people should be running build_wheel in that case. That approach would only make sense if you had to run some build step to generate the metadata, but not the whole build.

So, without access to build-time data METADATA, entry_points.txt, and license_files/ would be identical but WHEEL would not since build hooks can modify the tag/file name and whether or not it is ZIP-safe.

Can you elaborate here?

[ofek]

Can you elaborate here?

Hatch has the concept of build hooks like https://github.com/ofek/hatch-mypyc. Wheels recognize these fields which plugins can use to modify builds. For example, the aforementioned one does this to package compiled files.

You can just build the wheel in that case, the PEP even says that.

That's a really good point. @musicinmybrain Is there a reason why Fedora doesn't do that?

[pradyunsg]

3. We could actually build and cache the wheel here, and extract the real .dist-info directory. The build_wheel hook would then simply reuse the wheel from this stage.

That's what pip and pep517 do under the hood. As noted already, build also implements this logic (I don't know if it uses that via the CLI).

[musicinmybrain]

You can just build the wheel in that case, the PEP even says that.

That's a really good point. @musicinmybrain Is there a reason why Fedora doesn't do that?

I’m not sure I’m in a position to answer that question clearly, but I’ll try to rough out some context.

A modern Fedora RPM spec file for a Python library looks something like:

# [… name, description, sources, etc. …]
BuildRequires:  python3-devel

%package python3-foo
# […]

%prep
# unpack the tarball
%autosetup -n foo-%{version}

%generate_buildrequires
# -r means generate runtime dependencies, not just build-time, and is the default
%pyproject_buildrequires -r

%build
%pyproject_wheel

%install
%pyproject_install
%pyproject_save_files foo

%check
# run the tests

%files -n python3-foo -f %{pyproject_files}

The %prep section runs before %generate_buildrequires, then any additional build dependencies are installed based on its output (always as RPM packages, in an offline environment), then the wheel is built in %build and so on. Nothing else can be installed once %build starts.

Generating the runtime dependencies as (RPM-level) build dependencies prevents accidentally creating a package that can’t be installed due to unsatisfied dependencies, and it’s also typically necessary for running any tests.

Maybe there’s a way to be more clever in %pyproject_buildrequires, but on the other hand, the prepare_metadata_for_build_wheel hook, while optional per the relevant PEPs, is implemented in other build systems I’ve encountered. There was quite a bit of discussion about whether and how to support it in build; see also the associated PR.

Here is where and how the hook is actually used in Fedora.

A workaround to get the metadata I actually need to generate runtime dependencies for userpath looks like this:

BuildRequires:  python3dist(tomli)
# […]
%generate_buildrequires
'%{python3}' <<EOF
from tomli import load

def emit(tomlbase, reqtag, getdeps):
    with open(f'{tomlbase}.toml', 'rb') as cfgfile:
        deps = getdeps(load(cfgfile))
    with open(f'requirements.{reqtag}.txt', 'w') as reqfile:
        reqfile.writelines(f'{dep}\n' for dep in deps)

emit('pyproject', 'pyproject', lambda cfg: cfg['project']['dependencies'])
EOF
%pyproject_buildrequires -R requirements.pyproject.txt

Another workaround would be to simply maintain the list of BuildRequires manually by copying them out of pyproject.toml and checking them on each update. For a long time, that was how Python packages were handled in Fedora, but it has proved to be error-prone.

Mentioning @hroncok, who is the primary author of this machinery in Fedora and who might be interested in this discussion.

[hroncok]

The current situation is that if the prepare metadata for wheel hook is not available, our %pyproject_buildrequires macro cannot do anything wrt runtime requires. This could be changed by falling back to building the wheel instead, but that could lead to surprising results. For instance, the environment variables set during the %build phase are different than during the %generate_buildrequires phase. I'm sure there are semantic expectations about what is done in %generate_buildrequires and building the entire package isn't what RPM devs expected. Technically, it's probably possible.

[ofek]

I keep pondering whether or not to implement this but it just doesn't make sense to given that build hooks may modify the WHEEL file and PEP 517 explicitly outlines what to do in this case.

I think each distribution's machinery should adhere to the PEP. Is that okay?

[hroncok]

@musicinmybrain If you wish to experiment with building the wheel in %generate_buildrequires, open an RFE for pyproject-rpm-macros. I bet it's actually doable somehow.

[musicinmybrain]

@musicinmybrain If you wish to experiment with building the wheel in =generate_buildrequires, open an RFE for pyproject-rpm-macros. I bet it's actually doable somehow.

Thanks. Currently, there are three packages in Fedora that would benefit: python-hatchling, python-userpath, and hatch (once 1.0 is released). For now, I think that I’m personally happier maintaining the scripted workarounds in these packages than figuring out the implications and caveats of building the wheel in %generate_buildrequires in the general case.

If hatch/hatchling becomes quite popular, or if other build systems make the same choice not to offer this hook, then it might become much more beneficial to figure out how to deal with this in %pyproject_buildrequires.

I’ll go ahead and close this issue. Thanks for considering and investigating it.

[hroncok]

@musicinmybrain If you wish to experiment with building the wheel in %generate_buildrequires, open an RFE for pyproject-rpm-macros. I bet it's actually doable somehow.

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=2076994

tox 4 now uses hatchling

[lelit]

From the above I cannot understand the workaround, or otherwise why this was closed...

Trying to install platformdirs 2.5.2 in a poetry controlled project (under NixOS) I got the following:

Processing /build/platformdirs-2.5.2
  Running command Preparing metadata (pyproject.toml)
  Traceback (most recent call last):
    File "/nix/store/byxadvq5s849vl5xsla3rf2p2802syv3-python3.10-pip-22.0.3/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
      hook = backend.prepare_metadata_for_build_wheel
  AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_wheel'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/nix/store/byxadvq5s849vl5xsla3rf2p2802syv3-python3.10-pip-22.0.3/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/nix/store/byxadvq5s849vl5xsla3rf2p2802syv3-python3.10-pip-22.0.3/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/nix/store/byxadvq5s849vl5xsla3rf2p2802syv3-python3.10-pip-22.0.3/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/nix/store/g8sz7bxjqf3pwididawh9ngirsmqjxgr-python3.10-hatchling-0.22.0/lib/python3.10/site-packages/hatchling/build.py", line 41, in build_wheel
      return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
    File "/nix/store/g8sz7bxjqf3pwididawh9ngirsmqjxgr-python3.10-hatchling-0.22.0/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 103, in build
      configured_build_hooks = self.get_build_hooks(directory)
    File "/nix/store/g8sz7bxjqf3pwididawh9ngirsmqjxgr-python3.10-hatchling-0.22.0/lib/python3.10/site-packages/hatchling/builders/plugin/interface.py", line 318, in get_build_hooks
      raise ValueError('Unknown build hook: {}'.format(hook_name))
  ValueError: Unknown build hook: vcs

Thanks for any hint!

[ofek]

tox-dev/platformdirs@d6ab2ca#commitcomment-71548711

[lelit]

Thanks, will try adding also hatch-vcs...

[musicinmybrain]

Thanks, will try adding also hatch-vcs...

Packaging for Fedora Linux, I find that simply having hatch-vcs available is sufficient for a wheel built from the platformdirs PyPI sdist to be properly versioned.

I don’t know much about the NixOS Way of Doing Things, but for distribution packagers in general, if you find yourself needing to build from a GitHub archive instead of a PyPI sdist for some reason (because some upstreams don’t include documentation or tests in the PyPI sdist, for example, although this isn’t an issue for platformdirs), then it’s good to know that hatch-vcs uses setuptools-scm, so the trick

export SETUPTOOLS_SCM_PRETEND_VERSION="${MY_PACKAGE_VERSION}"

works for hatch-vcs too.

[ofek]

• Standard Python support dependabot/dependabot-core#3290 (comment)
• Implement minimal versions of metadata preparation methods #466