[Bug]: PEX warns about missing `setuptools` from `install_requires` in matplotlib

[georgipopovhs]

Bug summary

Using matplotlib with PEX warns about using namespace packages but missing setuptools from install_requires:

PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. matplotlib==3.5.2 namespace packages:
  mpl_toolkits

These distributions should fix their `install_requires` to include `setuptools`

This error is triggering the internal monitoring system (sentry) at our company.
matplotlib is installed as catboost dependency.
Interested in if this is a legit warning, and whether you would be open to including setuptools?

Code for reproduction

$ pip install pex
$ pex --version
2.1.92
$ pex matplotlib==3.5.2
/private/var/folders/t0/zb49xf010cq4mqzwwxw8f31h0000gp/T/tmpbwxjl1xt/.bootstrap/pex/environment.py:657: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. matplotlib==3.5.2 namespace packages:
  mpl_toolkits

These distributions should fix their `install_requires` to include `setuptools`
  "should fix their `install_requires` to include `setuptools`".format(dists=dists)
Python 3.7.9 (default, Mar 12 2021, 20:11:13)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

Actual outcome

Warning about missing setuptools from install_requires.

Expected outcome

No warning.

Additional information

No response

Operating system

Darwin Georgi.popov-6MNMD6R-hypersci 20.6.0 Darwin Kernel Version 20.6.0: Tue Oct 12 18:33:42 PDT 2021; root:xnu-7195.141.8~1/RELEASE_X86_64 x86_64

Matplotlib Version

3.5.2

Matplotlib Backend

'MacOSX'

Python version

3.7.9

Jupyter version

n/a

Installation

pip

[tacaswell]

The steps to do here are:

1. verify that we we do need setuptools as a run-time dependency if we have a namespace package
2. add it to requirements in setup.py
3. add link to docs as to why we need the dependency as a comment

[georgipopovhs]

1. verify that we we do need setuptools as a run-time dependency if we have a namespace package

Reading through the documentation for packaging and setuptools, there is no mention of the workaround that mpl_toolkits uses with the try-except block.

There is a recommendation in the packaging guide to depend on setuptools when in doubt:

If the presence of setuptools is a concern then the package should just explicitly depend on setuptools via install_requires.

I also tested installing matplotlib and using mpl_toolkits in a docker container without setuptools... and it appeared to be working, though only with later versions of pip. I opened an issue over at PEX to ask if the warning was legit: pex-tool/pex#1815

Also, I found the following transitional note on setuptools website:

Setuptools automatically calls declare_namespace() for you at runtime, but future versions may not. This is because the automatic declaration feature has some negative side effects, such as needing to import all namespace packages during the initialization of the pkg_resources runtime, and also the need for pkg_resources to be explicitly imported before any namespace packages work at all. In some future releases, you’ll be responsible for including your own declaration lines, and the automatic declaration feature will be dropped to get rid of the negative side effects.

During the remainder of the current development cycle, therefore, setuptools will warn you about missing declare_namespace() calls in your init.py files, and you should correct these as soon as possible before the compatibility support is removed. Namespace packages without declaration lines will not work correctly once a user has upgraded to a later version, so it’s important that you make this change now in order to avoid having your code break in the field. Our apologies for the inconvenience, and thank you for your patience.

This seems relevant to the mpl_toolkits approach, though I am not sure how it works in cases when there is no setuptools in the environment. I presume it's being "inherited for free" from somewhere, or using a vendored version of pkg_resources, similarly to pex and pip (speculating).

Although the matter isn't all black and white, it seems to me like mpl_toolkits is using a hacky approach that just happens to work, and I would prefer adding setuptools. WDYT?

[georgipopovhs]

The suggestion in the PEX thread was to silence PEX, because the import in matplotlib is opportunistic - it's imported only if available. This is why I am closing the issue.

I would still advise in favor of having a look into the transitional note on setuptools which seem to be dropping support for __init__.py files without declare_namespace().

[tacaswell]

https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#legacy-namespace-packages suggests that we should not be going it this way at all

Looking at the blame: 56a8541 came in ~2008 when I suspect we were on the cutting edge of this being a available (hence why it was try-excepted).

However from https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#creating-a-namespace-package

Warning While native namespace packages and pkgutil-style namespace packages are largely compatible, pkg_resources-style namespace packages are not compatible with the other methods. It’s inadvisable to use different methods in different distributions that provide packages to the same namespace.

and just above it

Use pkg_resources-style namespace packages. This method is recommended if you need compatibility with packages already using this method or if your package needs to be zip-safe.

I'm pretty sure that we are currently zip safe and should stay that way and we have had this try...except for 14 years, I'm not inclined to make any changes unless we identify actual breakage, not just a warning.