0.981 Introduces unnecessary build requirements

[dvzrv]

Bug Report

The changes in #13445 introduced a lot of additional build-system requirements for environments that use PEP517 build environments (e.g. pypa/build and pypa/installer). This finds application in downstream distributions of mypy (i.e. all Linux distributions are affected by this).

If my understanding of PEP0517, PEP0518 and PEP0621 are correct, then apart from setuptools and wheel none of the other entries should be listed and instead be moved to other places in the pyproject.toml file.

To Reproduce

Run the following in e.g. this repo in a clean environment (only build, installer, setuptools and wheel installed) in this repository with release-0.980 or above:

python -m build --wheel --no-isolation

Expected Behavior

The build process does not require test or runtime requirements to be installed.

Actual Behavior

* Getting build dependencies for wheel...
running egg_info
writing mypy.egg-info/PKG-INFO
writing dependency_links to mypy.egg-info/dependency_links.txt
writing entry points to mypy.egg-info/entry_points.txt
writing requirements to mypy.egg-info/requires.txt
writing top-level names to mypy.egg-info/top_level.txt
reading manifest file 'mypy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no directories found matching 'scripts'
no previously-included directories found matching 'docs/build'
no previously-included directories found matching 'docs/source/_build'
warning: no previously-included files found matching '.gitmodules'
warning: no previously-included files found matching 'CONTRIBUTING.md'
warning: no previously-included files found matching 'CREDITS'
warning: no previously-included files found matching 'ROADMAP.md'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files found matching 'action.yml'
warning: no previously-included files found matching '.editorconfig'
warning: no previously-included files found matching '.git-blame-ignore-revs'
warning: no previously-included files found matching '.pre-commit-config.yaml'
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'mypy.egg-info/SOURCES.txt'

ERROR Missing dependencies:
        types-psutil
        types-typed-ast<1.6.0,>=1.5.8
        types-setuptools

Your Environment

• Mypy version used: >= 0.981
• Mypy command-line flags: n/a
• Mypy configuration options from mypy.ini (and other config files): n/a
• Python version used: 3.10.8

[ethanhs]

These are required for building mypyc compiled wheels of mypy, so they are valid build-time dependencies. I'm not aware of any configurable way to make some build dependencies optional, so I don't think we can make them optional either.

[dvzrv]

These are required for building mypyc compiled wheels of mypy, so they are valid build-time dependencies. I'm not aware of any configurable way to make some build dependencies optional, so I don't think we can make them optional either.

But that is not the case for building the common wheels.

Hmm, so the change in #13445 introduced wheels being built with the current or a previous version of mypy? Circular dependencies should really be avoided :S (looks like this is configurable via setup.py)
On a related note: Why is it not enough to install custom build requirements using pip?

From a downstream distribution point of view: What are distributions supposed to do? Bootstrapping all of these packages poses a lot of extra work on distributions when updating the Python interpreter (e.g. from 3.10 to 3.11).
PEP517/PEP518 in itself already introduces a lot of bootstrapping overhead (see https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/thread/ZK57U5LPIDDJWPXWU5DSIYSI3EWUBGPP/).

[JukkaL]

Is this about bootstrapping the types-psutil, types-typed-ast and types-setuptools packages specifically? These are all stub-only/data-only packages (i.e. no executable code, not even Python), and they have no dependencies, so I'd expect them to be fairly easy to support, compared to many other packages.

[dvzrv]

@JukkaL hm, so would you recommend to always have mypy be a mypyc compiled extension?
In the current setup no other way of building a wheel is possible with PEP517 (unless one modifies the files).

Historically, we have abstained from starting to package typeshed packages, as it means even more packages to maintain and usually they are used in development environment where one can install them using mypy/pip (and frankly these files should really be maintained by their respective upstreams or be replaced by typing the projects, but I digress).

While I understand the efficiency and speed benefits of using mypyc (also for itself), please note, that from a distribution point of view this may lead to many more packages that need to be bootstrapped for each interpreter update (we are currently even still struggling to wrap our heads around the rebuild order), depending on how many other projects go down this route.
I'm not saying, that it can not be done, but even fully switching to PEP517/PEP518 workflows is still a work in progress for the thousands of Python packages we currently maintain and this is a whole lot of work...

[JelleZijlstra]

mypyc-compiled mypy is significantly faster than pure-Python mypy, so as a distributor, I'd want to distribute mypyc-compiled packages to help my users.

[dvzrv]

As mentioned above, I clearly understand the benefits.

However, coming from a setuptools-only setup (python setup.py build) to a PEP517/PEP518 setup (which is what I'm looking into currently), it is no longer possible to build the plain wheel and until we start packaging typeshed packages as well, this is a problem.
I will look into that, if this is indeed the preferred way to distribute mypy now.

I wonder if providing the default wheel package on pypi is still a good idea though, if you do not intend the project to be built that way at all.

[JukkaL]

I wonder if providing the default wheel package on pypi is still a good idea though, if you do not intend the project to be built that way at all.

It's our fallback option on less commonly used platforms for which we don't provide compiled wheels yet (e.g. non-Intel Linux).

We could vendor the typeshed stub packages needed to build mypy. That would add some friction to mypy development, but if that would make it easier to package mypy, it may be an acceptable compromise. Thoughts?

[AlexWaygood]

We could vendor the typeshed stub packages needed to build mypy. That would add some friction to mypy development, but if that would make it easier to package mypy, it may be an acceptable compromise. Thoughts?

I don't have a strong opinion either way -- but that would also have prevented #14037 from happening.

[dvzrv]

We could vendor the typeshed stub packages needed to build mypy.

If I understand the setup correctly, then the stub package versions should be equal to their respective counterparts, right?
Bundling the stub packages would then not really be beneficial, as it means, that they may drift (they may still drift apart, depending on how fast the stub package is updated after an update to the respective counterpart has been made).

[JukkaL]

If I understand the setup correctly, then the stub package versions should be equal to their respective counterparts, right?

The stub package used for building mypy must be compatible with the corresponding non-stub package, but only for the library features mypy actually uses. Most of the time using an older stub package is fine, since packages don't arbitrarily change the APIs in backward incompatible ways, but there is a small risk. In that case you could perhaps cherry-pick some changes from the mypy the repo to build an older mypy version.

[ethanhs]

it is no longer possible to build the plain wheel

Unless I'm missing something, I don't think this is true? If you run a PEP 517 build without telling mypy to use mypyc it should build a plain pure-python wheel (make sure to unset MYPY_USE_MYPYC). Installing extra dependencies into the environment isn't really a great solution, python setup.py ... is deprecated, and pep517 is supposed to become the way to build packages. So if we want to support building mypyc-compiled wheels with pep517 the only correct way to do is include all the dependencies.

As a suggestion for a simpler work-around instead of packaging all of mypy's dependencies, if you only want to build a pure Python package, you could always patch pyproject.toml to remove the things you don't need.

[hauntsaninja]

I'm -1 on vendoring the stubs.

I'd be happy to include a pyproject.toml.min file or something that contains minimal build requirements for a pure Python wheel, if that makes patching things easier for distributors.

[ethanhs]

I'm kind of -1 on vendoring these packages as well. Distributions are almost certainly going to want to package stub packages from typeshed eventually, especially as more projects adopt compilation with mypyc. (e.g. black will also run into this problem I imagine).

These packages are also rather stable (especially types-typed-ast, which we actually should be able to remove soon IIRC).

I think providing a pyproject.toml.min seems reasonable.