Document upper limit for setuptools_scm and other upgrading issues

[djhoese]

I have some projects that don't get a lot of activity (or don't have enough time dedicated to maintenance) and as such were still using setuptools_scm_git_archive. The recent 8.0 release removed the .utils module which was used by setuptools_scm_git_archive and is now resulting in:

...
        File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
          module = import_module(match.group('module'))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 940, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/tmp/pip-build-env-hcld8jiy/overlay/lib/python3.11/site-packages/setuptools_scm_git_archive/__init__.py", line 6, in <module>
          from setuptools_scm.utils import data_from_mime, trace
      ModuleNotFoundError: No module named 'setuptools_scm.utils'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

During pip install. Should projects using setuptools_scm add an upper limit to their setuptools_scm dependency to avoid issues like this in the future? Should setuptools_scm_git_archive have a patch release that doesn't include this import (if possible)? I'm actively going through removing the use of the git archive package from all of my packages that still used it, but it will take time and right now none of our users can install our packages.

[RonnyPfannschmidt]

It's tricky to handle that

For anything directly using it or maintained public plugins i directly reached out

[s0undt3ch]

Our CI builds just broke with the new release and we can't really do anything about it since the issue occurs in one of our dependencies

https://github.com/saltstack/salt/actions/runs/6251062159/job/16971960893?pr=64991#step:11:977

And the failure

https://github.com/saltstack/salt/actions/runs/6251062159/job/16971960893?pr=64991#step:11:1285

[s0undt3ch]

FYI, we are forcefully installing from source(not wheel files) on that job.

[RonnyPfannschmidt]

I'll take a look at either disabling the particular plugin or providing a shim

I strongly recommend managing build dependents yourself if you have a custom pipeline

[s0undt3ch]

I'll take a look at either disabling the particular plugin or providing a shim

I strongly recommend managing build dependents yourself if you have a custom pipeline

We are. But they are pip installed nonetheless, and if the dependency doesn't pin their build dependencies, then we're stuck at this place.

The link to the failure also shows another problem:

  Traceback (most recent call last):
    File "/tmp/pip-build-env-om5toxgi/overlay/lib/python3.10/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 53, in read_pyproject
      section = defn.get("tool", {})[tool_name]
  KeyError: 'setuptools_scm'

[djhoese]

While I work on updating my packages I've made a PR for setuptools_scm_git_archive that should fix this:

Changaco/setuptools_scm_git_archive#22

But that would require making a release for it.

[RonnyPfannschmidt]

@djhoese i can provide a shim with a warning

[RonnyPfannschmidt]

im scrapping the idea of a shim, but i'll block the plugin from the entrypoints as its in core now

[djhoese]

What does that mean "block the plugin"? Setuptools SCM just won't try to load it? Since we list it in our pyproject.toml as a build requirement this will still be an issue won't it? Or...I guess it won't be imported then so then it can't fail to import the removed utils module?

[RonnyPfannschmidt]

if the entrypoint is skipped it wont import (it also needs stuff in other modules)

[RonnyPfannschmidt]

i'll complete this tommorow moring as i need sleep before i can complete the release

[RonnyPfannschmidt]

i'll yank 8.0.1 due the git-arcival issue and release a working and regression tested 8.0.2 tomorow

[djhoese]

I think 8.0.0 is the problem. That's when .utils was removed, right?

[s0undt3ch]

I think 8.0.0 is the problem. That's when .utils was removed, right?

8.0.0 is already yanked

i'll yank 8.0.1 due the git-arcival issue and release a working and regression tested 8.0.2 tomorow

To clarify, you'll both yank, and release tomorrow, correct?

Not yank today and release tomorrow?

[ziyuanzhao2000]

I'm experiencing this issue this afternoon when pip installing a project managed with poetry. It's a package written by my collaborator, and I have no idea where in that package setuptools is used if poetry is used already. Anyway, I'm seeing exactly the same error, and I wonder if there's a temporary fix that I can use for now? Thanks!

[djhoese]

@ziyuanzhao2000 Probably not, but hard to tell unless you link to a build log or copy/paste the error you're getting.

[ziyuanzhao2000]

@ziyuanzhao2000 Probably not, but hard to tell unless you link to a build log or copy/paste the error you're getting.

Should I open a new issue for that or paste the link to my error log under this issue?

Anyway, here you go: https://pastebin.com/s8M6Q39F

Encountered this error when trying to install project using pip install -e .. Notably, there's no explicit dependence on setuptools or setuptools_scm in my pyproject.toml file, so I think it's a dependence of pip.

[djhoese]

Just to be sure, what is your project and it's dependencies? I don't see anywhere in the log which dependencies are being installed when it fails.

[ziyuanzhao2000]

@djhoese project is here and you can see dependencies in the pyproject.toml file. I've commented out pyqt from the [tool.poetry.dependencies] section. I've tried to add "setuptools>=60" and "setuptools-scm<8.0" in the [build-system] section to force using lower versions of setuptools_scm, but so far it still doesn't quite work. Full error log is pasted here.

[djhoese]

Oh great. It's vispy. That's my package. However, it looks like that is coming from napari and they have pinned the version of vispy to an old one (0.12.2). The newest release 0.14.0 fixes the setuptools_scm_git_archive dependency.

You'll either need to check out napari's github repository and see if they have a planned fix/update or wait for setuptools_scm (this project) to release a workaround (hopefully tomorrow).