upgraded to v4, lost versioning in github actions workflows

[mathematicalmichael]

Hi there!

I'm in need of some help. I was on the verge of ditching pyscaffold [edit: back to loving it] until I went through the docs one last time and noticed the changes in v4, namely that it would not require pyscaffold for setup. One of the big reasons I was drawn to this project was because of the automatic versioning tied to git history. I really liked being able to tie github actions to release versions and have true CD.

pyscaffold 3.3 was causing problems for "Dockerizing" my package in the cloud and I thought that the upgrade might get around these issues. The only way I was able to successfully build my image was to strip out some of pyscaffold, and then I lost the version numbers. If I build locally, I got version numbers correctly (I copied over my git folder into the image).

Docker aside, even though github actions does some magic with git cloning that I don't quite understand, on the last version of pyscaffold I was able to get proper version numbers when building the packages in the CI/CD pipeline.

I tried v4 of pyscaffold and I'm noticing that it always builds my package with 0.0.0 as the version in the pipeline now. https://github.com/mathematicalmichael/mud-examples/runs/2434672475#step:6:73 for example.

When I build locally I get the proper versions, so I'm suspecting something with git is the problem. How might I go about having versioning work in github-actions-like environments which mess with git history? Am I stuck on the previous version of pyscaffold if I want to do so? I suspect that if I can get v4 to work in github actions with proper version numbers, I can also get it to work in docker without having to copy my .git folder.

I have three branches where I'm experimenting: one where I attempt dockerization with the previous pyscaffold, feature/docker, another where I just experiment with pyscaffold v4: hotfix/update-pyscaffold and one where I try to merge the two branches with a revised dockerfile: https://github.com/mathematicalmichael/mud-examples/tree/feature/docker-pyscaffold that would be the dream + saving grace of pyscaffold.

I really hate the alternative of managing a VERSION file, and feel that the tie to git is inherently helpful, if not a pain for CI/CD environments.

[FlorianWilhelm]

Hi @mathematicalmichael and thanks for your explanations. You are right that PyScaffold 4 is no longer needed at setup time, so it's no longer a requirement as this was demanded by many people over time. Still, setuptools-scm which does the actual magic of determining the version number by looking it up in the git repo is still needed at setup time, i.e. when building, and there is no way around this. But this is actually no problem since only Github Action needs this to build your package and push it to some artifact store.

The problem you might have is that Github Actions does a shallow checkout of your git repo. You can do a "normal" full-depth checkout with:

    - name: Checkout Repo
      uses: actions/checkout@v2
      with:
        fetch-depth: 0  # avoids shallow checkout as needed by setuptools-scm

Checkout the github workflow for building and publishing PyScaffold itself as reference.

[FlorianWilhelm]

Okay, my bad, I looked into the wrong branch, it was main instead of update-pyscaffold.

Thanks for adding the debug lines, so we know now that the git repo itself is sane and v0.0.18-13-g056dcbc is the version that we would like to see. But python setup.py --version says: UserWarning: Unknown distribution option: 'use_scm_version' what means that setuptools-scm is not installed or not found at least.

It seems that we might have forgotten one thing in the update path from version 3 to 4. Could you add

# DON'T CHANGE THE FOLLOWING LINE! IT WILL BE UPDATED BY PYSCAFFOLD!
setup_requires = setuptools_scm[toml]>=5

within [options] in setup.cfg? That should do the trick. setup_requires is already a deprecated but works for the time being. The new way would be that you add the following file pyproject.toml in your repo.

[build-system]
# AVOID CHANGING REQUIRES: IT WILL BE UPDATED BY PYSCAFFOLD!
requires = ["setuptools>=46.1.0", "setuptools_scm[toml]>=5", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
# See configuration details in https://github.com/pypa/setuptools_scm
version_scheme = "no-guess-dev"

Since everyone is right now in a transitional phase from setup.py to pyproject.toml we did not include this in the update path from version 3 to version 4 since certain features do not work using pyproject.toml. After the Python ecosystem and setuptools has settled a bit more, we most likely will release version 5.0 of PyScaffold that gets completely rid of setup.py.

[mathematicalmichael]

whoa, interesting! thanks so much for the explanation and context. I wasn't aware setup.py was slated for deprecation. At work I've been trying to move projects to setup.cfg files but others keep adding stuff to setup.py, and I wasn't even aware of the pyproject.toml path.

https://github.com/mathematicalmichael/mud-examples/pull/44/checks?check_run_id=2449445357#step:7:73 verifies that setup_requires = setuptools_scm[toml]>=5 was the missing piece!

Thanks again so much. I assume that if I implement the lines into pyproject.toml I can get rid of the version_scheme arguments in my setup.py? Could I get rid of it altogether or not yet (which features are missing from toml?)?

[FlorianWilhelm]

My understanding is that you will only still need setup.py if your package includes compiled source code, like with Cython for instance. pyproject.toml just defines the build system in a declarative way and many tools use it to store their configuration in. Our pyproject.toml defines setuptools as build system (alternatives are poetry, flit, etc.) which uses setup.cfg to store its parameters.

I guess you would not need version_scheme in setup.py except someone uses setup.py directly to build the package. This might be really likely to happen if you still have it, so I would rather leave it in. There is right now much going on in the development of setuptools and PyScaffold will adapt to these changes to make all goodies available as easy as possible. My recommendation would be to wait until things have settled and we have PyScaffold v5 unless you want your time spent on build issues rather than coding your use-case ;-)

[mathematicalmichael]

thanks! I really appreciate your help. I'm looking forward to v5! Would you suggest implementing some of pyproject.toml's options into my project now or just waiting to create the file until later?

I really like the setuptools-scm functionality. +++ on that. Excellent work surfacing these features to people like myself, who generally are just trying to "get through" packaging and focus on the functionality itself.

[mathematicalmichael]

just for anyone stumbling across this in the future: TL;DR:

• for any build-related job (ie packaging tests), you need infinite fetch depth (fetch-depth: 0) for any jobs with a checkout step
• setup_requires = setuptools_scm[toml]>=5 needs to be included in setup.py if you're still using that in place of pyproject.toml
• alternatively, you can implement the pyproject.toml snippet shown above