Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LookupError: setuptools-scm was unable to detect version if [tool.setuptools_scm] is in pyproject.toml #1011

Open
befeleme opened this issue Jan 29, 2024 · 17 comments

Comments

@befeleme
Copy link

project: https://github.com/ionelmc/python-lazy-object-proxy/
download the source code, extract the tar archive and enter the folder: https://github.com/ionelmc/python-lazy-object-proxy/archive/refs/tags/v1.10.0.tar.gz

In pyproject.toml there is empty table [tool.setuptools_scm] defined.
In setup.py you can find the use_scm_version keyword argument with fallback_version explicitly defined.

setup(                                                                                                                  
      name='lazy-object-proxy',                                                                                           
      use_scm_version={                                                                                                   
          'local_scheme': 'dirty-tag',                                                                                    
          'write_to': 'src/lazy_object_proxy/_version.py',                                                                
          'fallback_version': '1.10.0',                                                                                   
      }, ...

Running SETUPTOOLS_SCM_DEBUG=1 python setup.py --version ends up with a LookupError:

DEBUG setuptools_scm.overrides dist name: lazy-object-proxy
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
DEBUG setuptools_scm.discover found ep EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.entrypoints EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
Traceback (most recent call last):
  File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/setup.py", line 76, in <module>
    setup(
  File "/usr/lib/python3.12/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 147, in setup
    _setup_distribution = dist = klass(attrs)
                                 ^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/setuptools/dist.py", line 496, in __init__
    _Distribution.__init__(
  File "/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
    self.finalize_options()
  File "/usr/lib/python3.12/site-packages/setuptools/dist.py", line 935, in finalize_options
    ep(self)
  File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/.eggs/setuptools_scm-8.0.4-py3.12.egg/setuptools_scm/_integration/setuptools.py", line 121, in infer_version
    _assign_version(dist, config)
  File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/.eggs/setuptools_scm-8.0.4-py3.12.egg/setuptools_scm/_integration/setuptools.py", line 56, in _assign_version
    _version_missing(config)
  File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/.eggs/setuptools_scm-8.0.4-py3.12.egg/setuptools_scm/_get_version_impl.py", line 112, in _version_missing
    raise LookupError(
LookupError: setuptools-scm was unable to detect version for /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

I comment out the [tool.setuptools_scm] table and run SETUPTOOLS_SCM_DEBUG=1 python setup.py --version again.
No error. Version is correctly set. Please note the found EntryPoint name set to pyproject.toml while in fact all this data is read from setup.py:

DEBUG setuptools_scm.overrides dist name: lazy-object-proxy
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.fallbacks FALLBACK 1.10.0
INFO setuptools_scm.version version 1.10.0 -> 1.10.0
DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found <ScmVersion 1.10.0 dist=0 node=None dirty=False branch=None>
DEBUG setuptools_scm.version scm version <ScmVersion 1.10.0 dist=0 node=None dirty=False branch=None>
DEBUG setuptools_scm.version config Configuration(relative_to='pyproject.toml', root='.', version_scheme='guess-next-dev', local_scheme='dirty-tag', tag_regex=re.compile('^(?:[\\w-]+-)?(?P<version>[vV]?\\d+(?:\\.\\d+){0,2}[^\\+]*)(?:\\+.*)?$'), parentdir_prefix_version=None, fallback_version='1.10.0', fallback_root='.', write_to='src/lazy_object_proxy/_version.py', write_to_template=None, version_file=None, version_file_template=None, parse=None, git_describe_command=None, dist_name='lazy-object-proxy', version_cls=<class 'packaging.version.Version'>, search_parent_directories=False, parent=PosixPath('.'))
DEBUG setuptools_scm.dump_version dump 1.10.0 into src/lazy_object_proxy/_version.py
1.10.0

We came across this issue while debugging a weird behavior of two Fedora's build tools: copr, where the build consistently fails and locally in mock with exactly the same environment, where no such issue arises, the version is read correctly. Could it be that the mere order in which files are read creates such discrepancies?

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

For the record, when we see a build failure in our build system:

  DEBUG setuptools_scm.config file pyproject.toml
  DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
  DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
  DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
  DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
  DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
  DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
  DEBUG setuptools_scm.discover found ep EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
  DEBUG setuptools_scm.entrypoints EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
  DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
  DEBUG setuptools_scm.config file pyproject.toml

While when it actually works in a different one:

  DEBUG setuptools_scm.config file pyproject.toml
  DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
  DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
  DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
  DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
  DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
  DEBUG setuptools_scm.fallbacks FALLBACK 1.10.0
  INFO setuptools_scm.version version 1.10.0 -> 1.10.0
  DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found <ScmVersion 1.10.0 dist=0 node=None dirty=False branch=None>

The sources should be identical, so I suspect it's the order of files on the filesystem or similar thing we cannot affect.

@RonnyPfannschmidt
Copy link
Contributor

what versions of setuptools and setuptools_scm are involved

the empty table should be removed upstream
i also recommend setting the pretend version var explicitly

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

This is setuptools_scm 8.0.4 with setuptools 68.2.2 -- I can test with a newer one...

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

I was gonna say that setuptools-68.2.2 blows up and setuptools-69.0.3 works correctly, but no, it's just flaky.

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

I reproduced with Python 3.12.1 and:

Package           Version
----------------- -------
packaging         23.2
pip               23.2.1
setuptools        69.0.3
setuptools-scm    8.0.4
typing_extensions 4.9.0

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

the empty table should be removed upstream

I don't know why they have that. It being empty makes me feel uncomfortable as well. However, at least it should error explicitly or behave consistently if that is the problem.

i also recommend setting the pretend version var explicitly

We can definitively do that when we build the package, but nevertheless, if fallback_version is supposed to work, I guess we should figure out what's happening here, no?

@RonnyPfannschmidt
Copy link
Contributor

Ah wait, it never should hit fallbacks for GitHub tarballs

Use pypi sdists

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

Ah wait, it never should hit fallbacks for GitHub tarballs

Why not? It says "fallback". I am confused.

Use pypi sdists

To clarify: we have a workaround available, we are not trying to sort out how to do this, we are trying to figure out what's going on here.

@RonnyPfannschmidt
Copy link
Contributor

The git archival filename of the project is incorrect, thus it's not found

So it's incorrect repo setup

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

What does "git archival filename of the project" mean? This happens on unpackaged tarball. If I rename the directory to lazy-object-proxy-1.10.0 it still happens.

@RonnyPfannschmidt
Copy link
Contributor

The error is in the git repo

That Tag will never produce a viable tarball

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

I am sorry. I have no idea what you mean.

There is a fallback version set. Sometimes, it is respected, sometimes it is not. I have no idea yet why. If I delete .git-archival.txt I still have the same error. If I clone the repo and delete .git, I have the same error.

What is the meaning of fallback_version, if not the fallback version?

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

For the record, the error does indeed tell me that what I do is wrong: "Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work."

However, things I don't understand:

  • Why does this work sometimes (no sure when yet)?
  • Why does this work when we delete pyproject.toml?
  • Why does it not fallback to the fallback version?

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

This is the smallest reproducer I can come up with. Has no git involved whatsoever.

I only have 2 files:

$ cat pyproject.toml 
[tool.setuptools_scm]

$ cat setup.py 
from setuptools import setup

setup(
    name='xxx',
    use_scm_version={
        'local_scheme': 'dirty-tag',
        'write_to': '_version.py',
        'fallback_version': '1.10.0',
    },
    setup_requires=['setuptools_scm'],
)

I create a venv and install latest setuptools and setuptools_scm into it:

$ python3.12 -m venv venv
$ venv/bin/pip install setuptools_scm setuptools
...
Successfully installed packaging-23.2 setuptools-69.0.3 setuptools_scm-8.0.4 typing-extensions-4.9.0

It works:

$ venv/bin/python setup.py --version
...
1.10.0

Now let me uninstall and install setuptools again. This should make no difference:

$ venv/bin/pip uninstall setuptools
...
  Successfully uninstalled setuptools-69.0.3

$ venv/bin/pip install setuptools
...
Successfully installed setuptools-69.0.3

Yet suddenly, it no longer works:

$ rm _version.py   # optional step, does not change the outcome
$ venv/bin/python setup.py --version
...
Traceback (most recent call last):
  File ".../scmscm/setup.py", line 3, in <module>
    setup(
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/_distutils/core.py", line 147, in setup
    _setup_distribution = dist = klass(attrs)
                                 ^^^^^^^^^^^^
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/dist.py", line 303, in __init__
    _Distribution.__init__(self, dist_attrs)
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
    self.finalize_options()
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/dist.py", line 654, in finalize_options
    ep(self)
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools_scm/_integration/setuptools.py", line 121, in infer_version
    _assign_version(dist, config)
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools_scm/_integration/setuptools.py", line 56, in _assign_version
    _version_missing(config)
  File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools_scm/_get_version_impl.py", line 112, in _version_missing
    raise LookupError(
LookupError: setuptools-scm was unable to detect version for .../scmscm.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

@hroncok
Copy link
Contributor

hroncok commented Jan 29, 2024

A script:

cat > pyproject.toml << EOF
[tool.setuptools_scm]
EOF

cat > setup.py  << EOF
from setuptools import setup

setup(
    name='xxx',
    use_scm_version={
        'local_scheme': 'dirty-tag',
        'write_to': '_version.py',
        'fallback_version': '1.10.0',
    },
    setup_requires=['setuptools_scm'],
)
EOF

python3.12 -m venv venv
venv/bin/pip install -Uq setuptools_scm setuptools

venv/bin/python setup.py --version  # works

venv/bin/pip uninstall -qy setuptools
venv/bin/pip install -q setuptools

venv/bin/python setup.py --version  # breaks

@RonnyPfannschmidt
Copy link
Contributor

That particular error is indeed unexpected

@jayaddison
Copy link

@hroncok thanks for that minimal repro case - I've been investigating a similar problem with the python-trx-python package in Debian, where a _version.py file is unpredictably created-or-not at package build-time -- it has a (non-empty) [tool.setuptools_scm] config table in pyproject.toml.

I believe the unpredictable behaviour is likely to be from the same cause, and that it relates to the order that the version_keyword and infer_version functions in setuptools_scm are invoked during package build.

Findings (so far)

Both the version_keyword and infer_version functions are declared as entrypoints in entry_points.txt file of setuptools_scm:

  • [project.entry-points."distutils.setup_keywords"]
    use_scm_version = "setuptools_scm._integration.setuptools:version_keyword"
  • [project.entry-points."setuptools.finalize_distribution_options"]
    setuptools_scm = "setuptools_scm._integration.setuptools:infer_version"

In the case of version_keyword, the function is placed in the distutils.setup_keywords group, handled by _finalize_setup_keywords in setuptools -- and therefore ultimately becomes part of the same finalize_distribution_options group where infer_version is found. This occurs indirectly, via the entry_points.txt file of setuptools, separate to the entrypoints file of setuptools_scm.

The finalize_distribution_options entrypoint group is a documented distribution-customization feature of setuptools.

Although the finalize_distribution_options group does allow functions to declare an order attribute on their callables to declare a runtime evaluation order, the default ordering is undefined (all items are defaulted to zero) and lacks a tiebreaker comparison.

Issue #1022 seems to relate to this area too.

Observations / possible solutions

What I find locally following installation of setuptools from recent tagged source versions (v69.2.0 for example) is that without changing any of the library code, I can get your minimal test case to pass/fail reliably by simply adjusting the version number in setuptools:setup.cfg and reinstalling it. That doesn't make much sense except to indicate a possible source of nondeterminism that does not relate to the code within setuptools itself, yet that can somehow be affected by (re)installations of the package. Ordering of reads from the filesystem, file timestamps, file paths and cache locations spring to mind as possible explanations, but I haven't confirmed the specific cause in this case yet.

I do think that the nondeterminism could be mitigated somewhere around the finalize_options function by creating a reliable tiebreaker for sort-ordering of the entrypoint functions, or allocating fixed order values for them, but I think it would be better to determine what the origin of the ordering difference is before patching over it.

netgate-git-updates pushed a commit to pfsense/FreeBSD-ports that referenced this issue Apr 26, 2024
- Add a pre-test stage that adds a [tool.setuptools_scm] section to
  project.toml, so that tests can detect the version.
  Otherwise 5 tests fail with an error similar to

________________________________________________________________________________________________________________________ test_fail_no_traceback ________________________________________________________________________________________________________________________

raise_on_session_done = <function raise_on_session_done.<locals>._func at 0x3f63e7392c10>, tmp_path = PosixPath(/tmp/pytest-of-root/pytest-5/test_fail_no_traceback0), capsys = <_pytest.capture.CaptureFixture object at 0x3f63e5c0aca0>

    def test_fail_no_traceback(raise_on_session_done, tmp_path, capsys):
        raise_on_session_done(ProcessCallFailedError(code=2, out="out\n", err="err\n", cmd=["something"]))
        with pytest.raises(SystemExit) as context:
            run_with_catch([str(tmp_path)])
        assert context.value.code == 2
        out, err = capsys.readouterr()
        assert out == f"subprocess call failed for [{something!r}] with code 2\nout\nSystemExit: 2\n"
>       assert err == "err\n"
E       AssertionError: assert  ... nerrn == errn
E         +                     WARNING  pyproject.toml does not contain a setuptools.py:119
E         +                              tool.setuptools_scm section
E           err

tests/unit/config/test___main__.py:52: AssertionError

See also pypa/setuptools_scm#1011

Release changes:	https://github.com/pypa/virtualenv/releases/tag/20.26.0
Reported by:	Bernát Gábor <notifications@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants