Python pins raise InvalidVersion error with setuptools 66.0.0

[duncanmmacleod]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

conda smithy regenerate is failing to regenerate feedstocks in an environment with setuptools>= 66.0.0:

$ conda run --name base conda smithy regenerate -c auto
Adding in variants from internal_defaults
Adding in variants from /tmp/tmpvfzp7avx/conda_build_config.yaml
Adding in variants from argument_variants

/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda/cli/conda_argparse.py:165: PendingDeprecationWarning: Loading conda subcommands via executables is pending deprecation in favor of the plugin system.
  warnings.warn(
INFO:conda_smithy.configure_feedstock:Downloading conda-forge-pinning-2023.01.17.14.17.22
INFO:conda_smithy.configure_feedstock:Extracting conda-forge-pinning to /tmp/tmpvfzp7avx
Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
WARNING:conda_build.config:Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
WARNING:conda_build.config:Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
INFO:conda_build.variants:Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from /tmp/tmpvfzp7avx/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from argument_variants
Traceback (most recent call last):
  File "/home/duncan/opt/mambaforge/bin/conda-smithy", line 10, in <module>
    sys.exit(main())
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_smithy/cli.py", line 670, in main
    args.subcommand_func(args)
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_smithy/cli.py", line 486, in __call__
    self._call(args, tmpdir)
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_smithy/cli.py", line 491, in _call
    configure_feedstock.main(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_smithy/configure_feedstock.py", line 2277, in main
    render_azure(env, config, forge_dir, return_metadata=True)
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_smithy/configure_feedstock.py", line 1389, in render_azure
    return _render_ci_provider(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_smithy/configure_feedstock.py", line 690, in _render_ci_provider
    metas = conda_build.api.render(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/api.py", line 38, in render
    metadata_tuples = render_recipe(recipe_path, bypass_env_check=bypass_env_check,
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/render.py", line 849, in render_recipe
    rendered_metadata = distribute_variants(m, variants,
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/render.py", line 711, in distribute_variants
    used_variables = metadata.get_used_loop_vars(force_global=False)
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/metadata.py", line 2558, in get_used_loop_vars
    for var in self.get_used_vars(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/metadata.py", line 2661, in get_used_vars
    meta_yaml_reqs = self._get_used_vars_meta_yaml(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/metadata.py", line 2746, in _get_used_vars_meta_yaml
    reqs_text, recipe_text = self._get_used_vars_meta_yaml_helper(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/metadata.py", line 2714, in _get_used_vars_meta_yaml_helper
    recipe_text = self.get_recipe_text(
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/metadata.py", line 2013, in get_recipe_text
    ns_cfg(self.config),
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/metadata.py", line 102, in ns_cfg
    defaults = variants.get_default_variant(config)
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/variants.py", line 109, in get_default_variant
    base.update(_get_default_compilers(config.platform, python))
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda_build/variants.py", line 90, in _get_default_compilers
    if parse_version(py_ver) >= parse_version('3.5'):
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '3.8.* *_cpython'

This is because this project uses complicated pins for Python itself, see:

conda-forge-pinning-feedstock/recipe/conda_build_config.yaml

Lines 651 to 655 in fbdfff4

	python:
	# part of a zip_keys: python, python_impl, numpy
	- 3.8.* *_cpython
	- 3.9.* *_cpython
	- 3.10.* *_cpython

This can be reproduced via:

$ mamba list setuptools
# packages in environment at /home/duncan/opt/mambaforge:
#
# Name                    Version                   Build  Channel
setuptools                65.6.3             pyhd8ed1ab_0    conda-forge
$ python -c "import pkg_resources; print(pkg_resources.parse_version('3.8.* *_cpython'))"
/home/duncan/opt/mambaforge/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 3.8.* *_cpython is an invalid version and will not be supported in a future release
  warnings.warn(
/home/duncan/opt/mambaforge/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
3.8.* *_cpython
$ mamba install -n base -y -q "setuptools>=66.0.0"
/home/duncan/opt/mambaforge/lib/python3.10/site-packages/conda/core/index.py:32: PendingDeprecationWarning: `conda.core.index.check_whitelist` is pending deprecation and will be removed in a future release. Please use `conda.core.index.check_allowlist` instead.
  warnings.warn(
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
$ mamba list setuptools
# packages in environment at /home/duncan/opt/mambaforge:
#
# Name                    Version                   Build  Channel
setuptools                66.0.0             pyhd8ed1ab_0    conda-forge
$ python -c "import pkg_resources; print(pkg_resources.parse_version('3.8.* *_cpython'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/duncan/opt/mambaforge/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '3.8.* *_cpython'

Is the easiest solution to have this feedstock pin to setuptools<66.0.0 until a workaround is integrated into conda-build or conda-smithy?

Installed packages

# packages in environment at /home/duncan/opt/mambaforge:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
anyio                     3.6.2              pyhd8ed1ab_0    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
blinker                   1.5                pyhd8ed1ab_0    conda-forge
boa                       0.14.0             pyhd8ed1ab_0    conda-forge
boolean.py                3.7                        py_0    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
chardet                   5.1.0           py310hff52083_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
coloredlogs               15.0.1             pyhd8ed1ab_3    conda-forge
commonmark                0.9.1                      py_0    conda-forge
conda                     22.11.1         py310hff52083_1    conda-forge
conda-build               3.23.3          py310hff52083_0    conda-forge
conda-forge-pinning       2023.01.17.14.17.22      hd8ed1ab_0    conda-forge
conda-package-handling    2.0.2              pyh38be061_0    conda-forge
conda-package-streaming   0.7.0              pyhd8ed1ab_1    conda-forge
conda-smithy              3.22.1             pyhd8ed1ab_0    conda-forge
conda-verify              3.1.1           py310hff52083_1006    conda-forge
cryptography              39.0.0          py310h34c0648_0    conda-forge
curl                      7.87.0               hdc1c0ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
deprecated                1.2.13             pyh6c4a22f_0    conda-forge
expat                     2.5.0                h27087fc_0    conda-forge
filelock                  3.9.0              pyhd8ed1ab_0    conda-forge
fmt                       9.1.0                h924138e_0    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
git                       2.39.0          pl5321h693f4a3_0    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.30             pyhd8ed1ab_0    conda-forge
glob2                     0.7                        py_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
humanfriendly             10.0            py310hff52083_4    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.0.0              pyha770c72_0    conda-forge
importlib_resources       5.10.2             pyhd8ed1ab_0    conda-forge
isodate                   0.6.1              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
libarchive                3.6.2                h3d51595_0    conda-forge
libcurl                   7.87.0               hdc1c0ab_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libimagequant             2.17.0               h7f98852_1    conda-forge
liblief                   0.12.3               h27087fc_0    conda-forge
libmamba                  1.1.0                hde2b089_3    conda-forge
libmambapy                1.1.0           py310h1428755_3    conda-forge
libnghttp2                1.51.0               hff17c54_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.23               h3eb15da_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
license-expression        1.2                        py_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.1.0           py310h51d5547_3    conda-forge
markupsafe                2.1.1           py310h5764c6d_2    conda-forge
msrest                    0.6.21             pyh44b312d_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
patch                     2.7.6             h7f98852_1002    conda-forge
patchelf                  0.17.2               h58526e2_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
perl                      5.32.1          2_h7f98852_perl5    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
prompt-toolkit            3.0.36             pyha770c72_0    conda-forge
prompt_toolkit            3.0.36               hd8ed1ab_0    conda-forge
psutil                    5.9.4           py310h5764c6d_0    conda-forge
py-lief                   0.12.3          py310hd8f1fbe_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h5764c6d_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pycrypto                  2.6.1           py310h6acc77f_1006    conda-forge
pycryptodome              3.16.0          py310h1419917_0    conda-forge
pygithub                  1.57               pyh1a96a4e_0    conda-forge
pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.6.0              pyhd8ed1ab_0    conda-forge
pynacl                    1.5.0           py310h5764c6d_2    conda-forge
pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.8          h4a9ceb5_0_cpython    conda-forge
python-libarchive-c       4.0             py310hff52083_2    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2022.7.1           pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.28.2             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rich                      13.1.0             pyhd8ed1ab_0    conda-forge
ripgrep                   13.0.0               h2f28480_2    conda-forge
ruamel.yaml               0.17.21         py310h5764c6d_2    conda-forge
ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
ruamel_yaml               0.15.80         py310h5764c6d_1008    conda-forge
scrypt                    0.8.18          py310ha302636_4    conda-forge
setuptools                66.0.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.40.0               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
urllib3                   1.26.14            pyhd8ed1ab_0    conda-forge
vsts-python-api           0.1.25             pyhd8ed1ab_1    conda-forge
watchgod                  0.8.2              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1          py310h5764c6d_1    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zipp                      3.11.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstandard                 0.19.0          py310hdeb6495_1    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

Environment info

active environment : base
    active env location : /home/duncan/opt/mambaforge
            shell level : 1
       user config file : /home/duncan/.condarc
 populated config files : /home/duncan/opt/mambaforge/.condarc
                          /home/duncan/.condarc
          conda version : 22.11.1
    conda-build version : 3.23.3
         python version : 3.10.8.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=5.15.79.1=0
                          __unix=0=0
       base environment : /home/duncan/opt/mambaforge  (writable)
      conda av data dir : /home/duncan/opt/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/igwn/linux-64
                          https://conda.anaconda.org/igwn/noarch
          package cache : /home/duncan/opt/mambaforge/pkgs
                          /home/duncan/.conda/pkgs
       envs directories : /home/duncan/opt/mambaforge/envs
                          /home/duncan/.conda/envs
               platform : linux-64
             user-agent : conda/22.11.1 requests/2.28.2 CPython/3.10.8 Linux/5.15.79.1-microsoft-standard-WSL2 debian/11 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[hmaarrfk]

might this be causing rerendering to fail locally too?

[duncanmmacleod]

might this be causing rerendering to fail locally too?

I'm not 100% sure what you mean by 'locally', but conda-smithy will fail in any environment that includes setuptools>=66.0.0. I spotted this locally first (on my machine), but haven't actually seen error reports from bot comments yet.

[jaimergp]

Yes, I am seeing this locally and in other non-conda-forge CIs. Although it seems to be more on the conda-build side.

[jakirkham]

The re-rendering bot rebuilds once weekly. So hasn't been affected yet. Still we should pin it.

Submitted PR ( conda-forge/webservices-dispatch-action#26 ) ( conda-forge/webservices-dispatch-action#27 ). There were other things need as well. So did them too.

We may want to do this other places depending on what all it affects

[jaimergp]

Confirmed this comes from conda-build

conda/conda-build#4732

I would repodata-patch it too. I'll open a PR.

[jakirkham]

I would repodata-patch it too. I'll open a PR.

This was just done with PR ( conda-forge/conda-forge-repodata-patches-feedstock#387 ). Will take a little bit before the patched repodata is available and in use on CDN

Edit: The repodata patch is live

[beckermr]

xref: conda-forge/conda-build-feedstock#201