Python: deprecate format and default to pyproject

[figsoda]

Motivation

Since the introduction of PEP 518 and some following PEPs that expands on the idea, the pyproject.toml format has gained a lot of adoption and became the de facto format for python packages.

However, a more popular option, and the current default format in nixpkgs is setuptools. Luckily, we might still be able to build these packages with format = "pyproject" because pyproject.toml falls back to using setuptools as its build backend as documented by pypa even if the package doesn't have a pyproject.toml. This might actually be preferred, because I ran into issues with format = "setuptools" when trying to package a setup.py package.

Proposal

I am proposing to deprecate the format option and replacing it with a pyproject option for packages that don't work with pyproject. We can do this over a longer period of time:

1. introduce a pyproject option that conflicts with the existing format option, where true is equivalent to format = "pyproject" and false is equivalent to format = "other"
   • python310Packages.buildPythonPackage: introduce pyproject option #254136
2. enforce newly added python packages to use the pyproject option instead of format
   • python310Packages.buildPythonPackage: introduce pyproject option #254136
3. convert existing format = "setuptools" and format = "pyproject" to pyproject = true and the rest to pyproject = false
4. deprecate format with a warning and make the default behavior pyproject = true (instead of format = "setuptools")
5. remove all pyproject = trues, as this is now the default
6. remove the format option altogether

Related Issues

• Python: remove format = "setuptools"; as default #124051
• pyproject build requires build-system definition nix-community/nix-init#119

[figsoda]

cc the python team: @FRidh @mweinelt @jonringer @tjni

[FRidh]

Yes, we should get rid of format, and this way forward sounds good. Although, I'd prefer if we right away consider also #170577 as that helps flattening our dependency tree, especially now since CA-derivations are well under way.

However, a more popular option, and the current default format in nixpkgs is setuptools. Luckily, we might still be able to build these packages with format = "pyproject" because pyproject.toml falls back to using setuptools as its build backend as documented by pypa even if the package doesn't have a pyproject.toml. This might actually be preferred, because I ran into issues with format = "setuptools" when trying to package a setup.py package.

We no longer use pip during build/install.

[pbsds]

A good chance to get rid of format = "flit".

I actually have a half decent pr to that end which I've yet to push. The only major blocker is the list of overrides defined by poetry2nix, which is massive.

[figsoda]

step 1: #254136

[FRidh]

A good chance to get rid of format = "flit".

I actually have a half decent pr to that end which I've yet to push. The only major blocker is the list of overrides defined by poetry2nix, which is massive.

I'd like us to remove format flit as soon as possible.

[pbsds]

I'd like us to remove format flit as soon as possible.

• python3Packages.flitBuildHook: remove #254309 😃

[mweinelt]

So format = "other" will be migrated to pyproject = false?

[figsoda]

So format = "other" will be migrated to pyproject = false?

Yes, format = "pyproject" | "setuptools" will become pyproject = true, and everything else will become pyproject = false

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-init-generate-nix-packages-from-urls-with-hash-prefetching-dependency-inference-license-detection-and-more/25035/18

[yajo]

Would it make sense to be able to point to the pyproject.toml file directly? So Nix could:

• Parse that TOML
• Read its build-backend.requirements
• If there's none, default to setuptools + wheel as PEP-518 explains.
• Try to find them by name from the current python3.pkgs

Hopefully most derivation definitions could be dramatically simplified.

[FRidh]

That would mean vendoring the pyproject.toml of projects in Nixpkgs since we do not want to have IFD.