Skip to content

Builds of merged PRs fail to upload to test.pypi.org, due to duplicate versions #288

@moreati

Description

@moreati
Contributor

Fallout from #287. Because the CI no longer applies a .devN suffix to the version then test.pypi.org rightly rejects the second package built with that <major>.<minor>.<patch>, e.g. https://github.com/bytecodealliance/wasmtime-py/actions/runs/15710411138/job/44266848706

Uploading distributions to https://test.pypi.org/legacy/
Uploading wasmtime-33.0.0-py3-none-android_26_arm64_v8a.whl
WARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Bad Request                                                            

Possible changes/fixes

  1. Reintroduce .devN, where N might be some combination of
    • number of commits from HEAD to initial commit (previous scheme)
    • number of commits since last major.minor.patch
    • commit hash
    • a build id provided by CI platform
    • timestamp of last commit
    • timestamp of build
    • arbitrary UUID
  2. Publish to test.pypi only when the version has not been previously uploaded
  3. Don't publish to test.pypi.org on merges

Considerations

  1. Any negative interactions with people installing directly from github repro (e.g. pip install git+https://github.com/bytecodealliance/wasmtime-py.git)
  2. Any impact on release workflow for project maintainers?

Initial discussion #287 (comment)

Activity

moreati

moreati commented on Jun 17, 2025

@moreati
ContributorAuthor

Trial: --build-number=<N>
Result: Negative/partial
Reason: Gets applied to wheel, but not to sdist

$ .venv/bin/python -m build -C="--build-option=--build-number" -C="--build-option=2"
...
adding 'wasmtime-33.0.0.dist-info/WHEEL'
adding 'wasmtime-33.0.0.dist-info/top_level.txt'
adding 'wasmtime-33.0.0.dist-info/RECORD'
removing build/bdist.macosx-11.0-arm64/wheel
Successfully built wasmtime-33.0.0.tar.gz and wasmtime-33.0.0-2-py3-none-any.whl
alexcrichton

alexcrichton commented on Jun 17, 2025

@alexcrichton
Member

The original intent of the test.pypi.org bits was to test all the platform name tags and such to ensure that pypi accepts everything and ensuring it's discovered before a release, but beyond that it's AFAIK not too useful to publish to test.pypi.org.

What about this: a manual-trigger-only github actions workflow (e.g. workflow_dispatch trigger) which publishes to test.pypi.org. That way we can maintain the version in-tree and put appropriate prerelease tags and such on it, and then when a dicy change comes in after merging I can trigger the workflow to publish to test.pypi.org and see if it passes. That would disable auto-publish but still enable publishes to happen in some cases.

moreati

moreati commented on Jun 18, 2025

@moreati
ContributorAuthor

I think auto-publish and auto-version are doable. I'm evaluating some more candidates today. I'll create a PR this evening or tomorrow.

moreati

moreati commented on Jun 18, 2025

@moreati
ContributorAuthor

Trial: https://setuptools-scm.readthedocs.io
Result: candidate
Discussion: Automatically generates/guesses version from source control (git) data. The version strategy and guessing behaviour are configurable. E.g.

  • 33.0.1.dev2+gab053ee
    • default settings
    • 2 commits since commit tagged 33.0.0, short commit hash ab053ee, no uncommitted changes
  • 33.0.0.post1.dev2+gbe99e94
    • version_scheme = no-guess-dev
    • 2 commits since commit tagged 33.0.0, short commit hash be99e94 no uncommitted changes
  • 33.0.0.post1.dev2+gbe99e94.d20250618
    • same as previous + dirty git work tree
wasmtime-py git:(issue288-version-scm) ✗ .venv/bin/python -m setuptools_scm          
33.0.1.dev2+gab053eewasmtime-py git:(issue288-version-scm) ✗ git log -1
commit ab053ee4447dd53ec23b90b6a77a8e36e0f8d909 (HEAD -> issue288-version-scm)
Author: Alex Willmer <alex@moreati.org.uk>
Date:   Wed Jun 18 07:48:29 2025 +0100

    WIP

Notes:

  • can write version to a (Python) file
  • AFAICT cannot read base version from a file, always uses SCM data
  • Probably the most widely used/established setuptools plugin that does versions
  • May also eliminate size of/need for MANIFEST.in, sdist inclusion is also driven by which files are checked in.
moreati

moreati commented on Jun 18, 2025

@moreati
ContributorAuthor

Trial: https://setuptools-git-versioning.readthedocs.io
Result: Candidate, leaning to recommended
Discussion: The recommended use pattern is file based, meaning the under-development version (e.g. 33.1.0) is stored in a plain text file, manually updated. A suffix is automatically added to this version, to indicate pre-releases and/or dirty builds. The closest variant to wasmtool-py's workflow appears to be https://setuptools-git-versioning.readthedocs.io/en/stable/schemas/file/dev_release_file.html. Example usage, with fictitous version 0.50.0

  1. HEAD (45fd99a) has configured and populated the new VERSION file, the revision is untagged, working directory is clean with no untracked files.
    setuptools-git-versioning generates the version 0.50.0.dev0, where

    • 0.50.0 has been read from VERSION
    • dev0 counts the numer of commits since VERSION was modified
    • a devN suffix is present because this commit is untagged
    wasmtime-py git:(issue288-version-git-versioning) git log -3 --oneline 
    45fd99a (HEAD -> issue288-version-git-versioning) Try setuptools-git-versioning
    1d44dad (upstream/main, main) Use Python "build" package as build frontend, don't directly run setup.py (#287)
    31c3457 (tag: 33.0.0, origin/main, origin/HEAD) Update to Wasmtime 33 (#281)wasmtime-py git:(issue288-version-git-versioning) cat VERSION 
    0.50.0wasmtime-py git:(issue288-version-git-versioning) .venv/bin/python -m setuptools_git_versioning
    0.50.0.dev0
  2. Tagging this commit indicates a release revision, removing the devN suffix.

    wasmtime-py git:(issue288-version-git-versioning) git tag 0.50.0wasmtime-py git:(issue288-version-git-versioning) .venv/bin/python -m setuptools_git_versioning
    0.50.0
  3. A dirty worktree adds a "local" suffix to the version. Distributions with such a suffix are reject by pypi.org

    wasmtime-py git:(issue288-version-git-versioning) touch not_checked_in.txtwasmtime-py git:(issue288-version-git-versioning) ✗ .venv/bin/python -m setuptools_git_versioning
    0.50.0.post0+git.45fd99a2.dirty
alexcrichton

alexcrichton commented on Jun 18, 2025

@alexcrichton
Member

Yeah that seems reasonable to me!

moreati

moreati commented on Jun 18, 2025

@moreati
ContributorAuthor

PR underway.

added 4 commits that reference this issue on Jun 18, 2025
0067008
9681a31
804cc50
a6f014a
added a commit that references this issue on Jun 20, 2025
7d53651
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @alexcrichton@moreati

      Issue actions

        Builds of merged PRs fail to upload to test.pypi.org, due to duplicate versions · Issue #288 · bytecodealliance/wasmtime-py