Lock files don't mesh with dynamic package version info

[hynek]

Since tox-uv just shipped lock files, I took a stab at moving attrs to a fully-locked dev environment.

Here's the experiment PR: python-attrs/attrs#1349

A problem I've run into is that packages often use dynamic packaging data that is based on git metadata (setuptools-scm, hatch-vcs, etc). As such, attrs's package version changes after every commit, which allows us to upload to test PyPI and continually verify our package. See, for example, https://test.pypi.org/project/attrs/24.2.1.dev19/

Unfortunately the current project gets locked like this:

[[package]]
name = "attrs"
version = "24.2.1.dev20"
source = { editable = "." }

Which means it's outdated after each commit, because every commit increments the number behind dev.

Is there a way around this that I've missed? If not, could there be built one? 😇 AFAICT, this is currently the biggest blocker for FOSS use from uv's side. I'm also not sure what the point of that version lock is in the first place for the current, editable project? But that's a topic for another day. :)

---

❯ uv --version
uv 0.4.12 (2545bca69 2024-09-18)

[sbidoul]

@hynek have you seen tool.uv.cache-keys ?

[hynek]

I have while searching this bug tracker, but I couldn't find a way for it to affect uv.lock? Changing the value to cache-keys = [{ git = true }, { file = "pyproject.toml" }] at least doesn't do anything when running uv.lock.

From the documentation (and its name 🤓) I've gathered it's mostly about how caching, not about locking? And TBH it sounds to me as if adding git as a git key, I'm actually adding cache busting.

[charliermarsh]

I think you could pass --upgrade-package attrs to ensure that attrs is rebuilt and that the lockfile is updated. Or even set it in your configuration:

[tool.uv]
upgrade-package = ["attrs"]

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. I don't know how to solve this holistically. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

[hynek]

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

Yeah that's what I meant in my last sentence, but I'm always assuming Chesterton's Fence. :) Setting it to dynamic seems the best way since it maps nicely to project.dynamic. Just dunno if it would be a problem that there's a case where there's an invalid version "number" in the field which would speak for omitting.

[jkeifer]

I too have this problem. I generally subscribe to the philosophy that I use a VCS for versioning, thus what is in the VSC repo shouldn't track it's own version because that's what I'm using the VCS for. Said another way, at best a version tracked within repo is imperfect for exactly the reasons described above about how you can't update a version from a commit without making another commit, i.e., a new version.

I'd love to understand the rationale behind storing this version in the lock file. Can anyone elaborate on that?

[burgholzer]

We are also facing a similar problem over at cda-tum/mqt-core#706 where we use renovate to keep the uv lock file up to date, which would be pretty convenient in principle.
However, currently it triggers a never ending chain of renovate update PRs because every update to the lock file adds a commit, which changes the setuptools-scm version, which causes another update PR to be triggered. (see cda-tum/mqt-core#703, cda-tum/mqt-core#704, cda-tum/mqt-core#705, cda-tum/mqt-core#706)

[LordFckHelmchen]

This problem is not only for dynamic versioning. We have a release-please workflow, which will create a PR for the next release. This will update the pyproject.toml version correctly but of course does not know about the uv.lock version.
If there would be an option to omit the project version from the uv.lock it would be great, as I currently also don't understand why it needs to be kept in the lock-file itself.

[david-waterworth]

As I mentioned in ttps://github.com//issues/7994 I usepython-semantic-release which behaves the same. The way I work around the infinite build loop is to configure my Jenkins script to skip commits that contains the python-semantic-release message - these commits include the changed pyproject.toml and __version__.py file, along with a generated CHANGELOG.md

As a workaround, I also added uv lock at the start of the build script and git add uv.lock at the end - this ensures that python-semantic-release commit includes the correctly versioned uv.lock file (I'm not sure yet if this causes other problems, i.e. updating other packages - ideally there would be a way of only updating the package version in uv.lock and nothing else.

[alejandrodnm]

I've just encountered this. We are also using release-please and uv in an internal project, after a release then the surprise is that the uv.lock was outdated.

Ours plans were to add uv to https://github.com/timescale/pgai but since we are already using release-please, we are hesitant due to this issue.

[RazerM]

While the setuptools-scm case seems difficult to solve, I think another common reason¹ a dynamic version is used is when the version is statically defined but not in pyproject.toml. Typically as a __version__ attribute like in this example:

1. Create example project

   uv init --lib mpypkg && cd mpypkg

2. Configure pyproject.toml to use a dynamic version

   cat <<EOF > pyproject.toml
   [build-system]
   requires = ["setuptools"]
   build-backend = "setuptools.build_meta"
   
   [project]
   name = "mpypkg"
   requires-python = ">=3.12"
   dynamic = ["version"]
   
   [tool.uv]
   cache-keys = [{ file = "pyproject.toml" }, { file = "src/mpypkg/__init__.py" }]
   
   [tool.setuptools.dynamic]
   version = { attr = "mpypkg.__version__" }
   
   [tool.setuptools]
   package-dir = { "" = "src" }
   
   [tool.setuptools.packages.find]
   where = ["src"]
   EOF

3. Add __version__ attribute to __init__.py

   cat <<EOF >> src/mpypkg/__init__.py
   __version__ = "1.0"
   EOF

4. Run uv lock

5. Change version number

   sed -i.bak 's/__version__ = "1\.0"/__version__ = "1.1"/' src/mpypkg/__init__.py

6. Run uv lock

What I expected:

 version = 1
 requires-python = ">=3.12"
 
 [[package]]
 name = "mpypkg"
-version = "1.0"
+version = "1.1"
 source = { editable = "." }

What happened: uv.lock still has version = "1.0"

I've also tried uv lock --no-cache --refresh to no avail.

The version doesn't change until I do an unrelated uv add or something. This seems like a cache invalidation problem more than anything else, which I expected cache-keys to solve.

---

❯ uv --version
uv 0.4.30 (61ed2a236 2024-11-04)

Footnotes

1. sometimes for backward compatibility reasons or in my case I can't move version into pyproject.toml until some other tooling I use catches up. ↩

[charliermarsh]

We can fix this, but I still strongly recommend against setting the version dynamically like that.

[charliermarsh]

I've fixed the issue @RazerM pointed out in #8867. I guess I'll leave this issue open though since it's more about the inherent incompatibility between lockfiles and VCS-based versioning.

[cpascual]

Hi @charliermarsh, you mention "inherent incompatibility of lock files and VCS-based versioning" , but you also mentioned:

We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

To me the above workaround seems good enough, specially if it is controlled by an opt-in configuration (e.g. omit_lock_version=["mypackage",]).

Is there any fundamental problem with this approach?

[cpascual]

my use case is that I use setuptools-scm and the presence of commit-dependent version numbers in uv.lock is a frustrating cause of git merge conflicts preventing what otherwise would be an automated merge.

[ceejatec]

We can fix this, but I still strongly recommend against setting the version dynamically like that.

@charliermarsh Out of curiosity, why?

[Coruscant11]

I just discovered this issue with the recent release 0.5.0. Using --locked when syncing the project makes everything fails. I have a dynamic version based on git tags, and when we release we tag our repository. So the lock file can be outdated without any code change.

I think excluding editable dependencies in the lock file could be a good workaround. Or we can also write "dynamic" as a version. By the way I will check what tools like poetry for example are doing in this use case.
@charliermarsh Is there any news on this? In the meantime, we will pin our version to the 0.4.30.

By the way I would suggest to list it in the breaking changes of the release

Thanks !!!

Edit: Forgot to mention, I was wondering if you already have some implementation ideas, I totally volunteer to work on this if needed. I will try a draft pull-request on my side for fun

[Stealthii]

We can fix this, but I still strongly recommend against setting the version dynamically like that.

If references or opinion articles exist around this topic, they are few in number. I think a few on this issue would be interested in wider discussion around this. PEP 621 introduced toml metadata, and as of writing the official specification still allows version to be set dynamically, with no recommendation to the contrary.

It's a pretty standard paradigm (see hatch-vcs, setuptools-scm, poetry-dynamic-versioning) for single project repos where versioning, point release branches, development builds, etc. are all managed and controlled via VCS change control.

Whilst alternatives exist for separating static versions (such as 0.5.1 from a generated identifier ref like f399a5271 in uv 0.5.1 (f399a5271 2024-11-08)), there are some compelling reasons to use dynamic (such as dunamai) formed versions:

• Unique reference of all release, post, dev, dirty distributions of Python packages
• All tarball/wheel builds being uniquely available in an internal PyPI repository for further testing or triage
• Rules around versioning and releases can be set by project owners irrespective of development language
• No room for user error (such as forgetting to update a static version of 0.5.1 for the git tag v0.5.1 and publishing builds)
• Exposing the version in test logs, headers, etc. automatically shows the exact build used, aiding debugging

---

My expectation for the project package that has a dynamic version (source = { editable = "." } specifically) would be simply not defining version in uv.lock for the local package (or providing a dynamic reference). This by nature isn't a lockable reference (especially when set by VCS signatures) whereas other uses of uv lock such as multi-project mono repos or static versions uphold the current behaviour.

It's worth noting that although version is defined dynamically in source control, builds of the package provide the version statically:

❯ ls -1 dist/*
dist/mypackage-0.5.0.dev5+g5d056c1-py3-none-any.whl
dist/mypackage-0.5.0.dev5+g5d056c1.tar.gz
❯ unzip -p dist/*.whl '*.dist-info/METADATA' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1
❯ tar -Oxf dist/*.tar.gz '*.egg-info/PKG-INFO' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1

[charliermarsh]

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

[charliermarsh]

...simply not defining version in uv.lock for the local package (or providing a dynamic reference)

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

[Coruscant11]

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

Yes this is what I saw this week-end when I was trying to contribute on this 🥲
But anyway, I had the opportunity to deep dive in the code base so it's fine.
We will have to trust you on this 😄 (No need to worry at all then)

Thanks!

[ceejatec]

it breaks the fundamental assumption that packages have versions

Not exactly, I don't think. The package always DOES have a version. It's just that the version may change outside of uv's immediate notice. Perhaps it would be fine if uv lock and uv sync and similar commands re-assessed the version if it's marked dynamic in the pyproject.toml. That wouldn't handle cases where the dynamic version contains the git commit SHA in some form, but I suspect that's pretty much an intractable problem.

That said, I feel like a cleaner solution would be to omit the main package from uv.lock entirely. However I'm definitely not familiar with the implementation here, so I don't know what unforeseen consequences that might have.

[hynek]

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

By the way, this can be inverted and made more standard-compliant while doing so by putting something like this into your __init__.py:

def __getattr__(name: str) -> str:
    if name != "__version__":
        msg = f"module {__name__} has no attribute {name}"
        raise AttributeError(msg)

    from importlib.metadata import version

    return version("YOUR-PACKAGE")

You get static metadata with no duplication.

[ninoseki]

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

• hatch-vcs: 2.2k files
• poetry-dynamic-versioning: 1.6k files

[roteiro]

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

• hatch-vcs: 2.2k files
• poetry-dynamic-versioning: 1.6k files

add to that:

• versioningit 420 files

[jamesbraza]

I think #8867 released in uv==0.5.0 leads uv to correctly respect dynamic metadata, which now breaks setuptools-scm + pre-commit users using the uv-lock hook: astral-sh/uv-pre-commit#25.

This is textbook Hyrum's Law 😆

---

To share on the SCM versioning debate, I use setuptools-scm as it's a nice way for an automated (no human error) workflow of GitHub tagged release to PyPI publish with matching version.

[charliermarsh]

Yeah, it seems like fundamentally the only way to make this work with setuptools-scm is to omit the version or otherwise mark it as dynamic. I don't think it's impossible, but it's not, like, an hour of work. I'm also fairly worried about how we would integrate this if PEP 751 gets approved -- the PEP doesn't make any carveouts for dynamic versions, and there hasn't been any discussion of it in the PEP thread. If we add a feature like this here, and it doesn't make it into the PEP, we'll have to remove it later.

[apollo13]

I think #8867 released in uv==0.5.0 leads uv to correctly respect dynamic metadata, which now breaks setuptools-scm + pre-commit users using the uv-lock hook: astral-sh/uv-pre-commit#25.

Can confirm that our workflow (using hatchling as build backend with a dynamic version) breaks when trying to do uv sync --locked since uv 0.5. 0.4.30 is fine. Whether it is actually #8867 that broke it I cannot tell, but sounds plausible.

[charliermarsh]

It likely was #8867 that led to the change, though sadly that really was a bugfix -- the existing behavior was wrong. Right now there just isn't a clear way to use scm-based versioning with lockfiles, but I'm thinking about it. My biggest hesitation to investing in it deeply is the aforementioned PEP 751. I don't want to do something that will become invalid once the standard exists.

[dsully]

Is @brettcannon aware of this problem? Should PEP-751 take this usage into account?

(I'm also not 100% sold we need a universal lockfile format. I believe the 715 discussion thread also mentions this. 🤷 )

[charliermarsh]

I've been deeply involved in the thread and don't think it's come up, but I'll post about it today.

(Edit: @hynek already did.)

[apollo13]

(I'm also not 100% sold we need a universal lockfile format. I believe the 715 discussion thread also mentions this. 🤷 )

Having a universal lockfile seems better than trying to add support for 100 different lock file formats to every other SBOM tool out there :D

[brettcannon]

Having a universal lockfile seems better than trying to add support for 100 different lock file formats to every other SBOM tool out there :D

I'm going to ask that if @dsully or anyone else wants to express concern over standardizing lock files they bring it up on the PEP 751 discussion over on discuss.python.org and not here as I don't think the uv issue tracker is the best place to talk about PEPs that affect the wider Python packaging ecosystem 😉

[hynek]

For posterity: https://discuss.python.org/t/pep-751-now-with-graphs/69721/86

[dsully]

Will do @brettcannon, thanks! Though looks like Hynek got there first 😎

[tomrutter]

Sorry to pipe in on a largely completed debate but I have one idea to put.

My issue here seems the same as many other users:

1. My python package uses automatic versioning based on git tag for fewer user errors and easier tracking of built artefacts.
2. The package version is included in lock file, which when committed changes commit tag and thus the version.
3. As part of CI, I check lock file is up to date (for reproducibility). This check fails because of the version change of the local workspace package.

Changing any one of these links could fix the issue. I think most of the discussion has been about 1 and 2. That is, either moving away from dynamic versioning or changing how lock files work for local packages.

However, a simpler resolution might be via step 3? If uv were to allow users to check that lock files are up to date apart from the versions of any current workspace packages that have dynamic versions then the current workflow would be able to continue. Effectively this returns to the workflow of the pre 0.5.0 uv, while still correctly updating the lock file (and re-installing the package). I would expect this would catch most of the current use cases for uv lock --frozen

This could work by adding an option to uv lock --frozen etc to allow versions of local packages with dynamic versioning to change but to error if any other dependencies change.

[charliermarsh]

I think the solution here will ultimately be to stop storing versions for source trees (either entirely, or when dynamic -- I prefer doing so entirely, and not based on whether it's dynamic). I just want PEP 751 to play out a little more before I make any change.

[charliermarsh]

(Once we have clarity on what we want to do, I'll prioritize the change.)

[nathanjmcdougall]

As a temporary kind of hacky semi-workaround, I am using uv export via uv-pre-commit. Changes to the dependencies in pyproject.toml get reflected in a changed requirements.txt file, failing the hook. Since the requirements.txt file doesn't include the version of the editable-installed package, it doesn't suffer from the same issue as checking against uv.lock directly.

In theory, the user could still get around this by modifying both the pyproject.toml and requirements.txt files but not the uv.lock file, since the hook is not checking the lockfile directly. I don't really think that would be possible in most cases without creating a requirements.txt file that mismatches what would get created by uv export so it would still fail, forcing a lockfile update.

It protects against a situation where a dependency gets declared in pyproject.toml but the user forgets to run uv sync or uv lock - the pre-commit would fail, and the lockfile would be updated as a side-effect. But if the user is using uv add then this adds little benefit since it's unlikely the user would declare in pyproject.toml manually in the first place.

My plan is to get my organization to stop using dynamic versioning in packages...

[mjpieters]

We use a Taskfile.yml task definition, so I have a little more flexibility, and I use it to apply an ugly work-around: When we update the lock I force the version to be 0.0.0 by setting SETUPTOOLS_SCM_PRETEND_VERSION (which works for either hatch-vcs or setuptools-scm):

  dev:lock:
    aliases:
      - lock
    desc: Lock dependencies
    preconditions:
      - sh: type "uv" &>/dev/null
        msg: Please install uv, see https://docs.astral.sh/uv/getting-started/installation/
    sources:
      - pyproject.toml
    generates:
      - pyproject.toml
      - uv.lock
    env:
      # avoid locking with a dynamic (hatch-vcs) version.
      SETUPTOOLS_SCM_PRETEND_VERSION: 0.0.0
    cmds:
      - uv lock

and then, when linting:

  dev:lint:
    aliases:
      - lint
    desc: Runs linters
    preconditions:
      - sh: type "uv" &>/dev/null
        msg: Please install uv, see https://docs.astral.sh/uv/getting-started/installation/
    cmds:
      - |
        SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0 uv lock --locked 2>/dev/null || {
          echo -e '\033[0;31mThe lockfile at `uv.lock` needs to be updated. To update the lockfile, run `task dev:lock`\033[0m'.
          exit 1
        } >&2

This lets us at least avoid recording the current dynamic version in the lock file.

[thcrt]

Just checking- is there any way to use dynamic versioning and build with GitHub Actions while also including uv.lockin version control?

I haven't been able to get it working, and I'm not sure whether this is a problem with actions/checkout, with hatch-vcs, with setuptools-scm or with uv.

I can run uv build on my own machine just fine, and when I do this after having tagged the latest commit, the resulting builds have the version of the tag. However, I've found it impossible to get the same thing to happen via a workflow. The uv sync step always seems to set the version to include a developmental release segment and a local version identifier, such as 0.2.2.dev0+g38cf6c2.d20241210.

The only way that I can find to prevent this from happening is removing uv.lock from version control. Obviously, this isn't ideal.

I'd love to know if there's something I'm missing, and it's possible to use hatch-vcs, uv and GitHub Actions workflows together. If this is an issue with one of the other projects involved, let me know and I'll report it to them :)

[nathanjmcdougall]

@thcrt This probably isn't exactly what you're looking for, but I've found this works:

https://gist.github.com/nathanjmcdougall/42135563c719cfda59a1de317da5116b

Note that this deliberately avoids running uv sync, to avoid the issue. I'm using hatch-vcs.

[thcrt]

@nathanjmcdougall Incredible. You are a saint (and a fellow Kiwi!)

Genuinely, thank you so much, this has been a major headache for me in two different projects.

I imagine that not running uv sync in CI may have its downsides, but I'm happy to run it locally and manually if that's the tradeoff for having builds work properly.

Thank you so much again.