Would you consider miniver for version management?

[hmaarrfk]

I know the versioneer transition was somewhat painful for many that relied on it, but it would help me work with your development version and report bugs more precisely.

https://github.com/jbweston/miniver

I can make the PR to enable it.

For those not familiar with versioneer:

The advantage is that given that you use tags for your releases, and a linear development stype (i.e. no support branches) you could have the current main's version be

$ git describe --tags
v0.1.18-10-g34b26ac

would report as

pygfx.__version__
'0.1.18.post10+g34b26ac'

or something like that.

let me know.

For me, i could start to add compatibility shims to my code that uses your repo without try except statements!

[almarklein]

I'm open to the idea. @Korijn ?

[almarklein]

Can you explain more about the problem that this would solve?

One advantage is being able to just push a tag without needing commits like #725.

Seeing #726, I also wonder if adding a get_full_version() function (or something like it) would also solve the problem, but in a simpler way. It could simply take the release version number (what we have now) and then adds a suffix calculated from git.

[hmaarrfk]

It could simply take the release version number (what we have now) and then adds a suffix calculated from git.

This hybrid approach is interesting. I'll try to give it a go when I have a second again. One constraint is to try to avoid the use of subprocess during a released version to ensure good import speed.

I also wonder if adding a get_full_version() function (or something like it)

I feel like package_name.__version__ is the amount of cognitive load that I can bear when trying to identify the version of the code I'm running. But I'm happy to get to a middle ground.

Ideally, this is something you build once, and lasts 3-4 years so I'm not willing to give up on a good solution just yet.

@Korijn brought up complexity, which is a real concern, when versioneer broke a few years back (the code was copied into many repos), it caused alot of churn for packing efforts.

Can you explain more about the problem that this would solve?

3 Aspects:

1. Downstream developers.
2. Automation of small tasks that add up cognitive load.
3. Faster resolution time for bugs reported by end users

Downstream code

The usecase is that in my downstream code I could for example start to be "compatible" with version 0.1.19 even before it comes out.

Contributing users (this can be you in your own downstream project too) can start to have the conditions:

if Version(pygfx.__version) > Version('0.1.18');
    print("do something special")

Automation of routine tasks

Improved release process cannot be understated. For me, I release on a daily basis so the delay in the process:

1. agreeing to do a release.
2. Writing the release notes (we use towncrier for this)
3. Create a PR with the release notes
4. Waiting for the CIs to finish.
5. Merge.
6. Tag (write the release again)
7. Then realize you forgot to bump the version number
8. Restart 1 through 6.
9. Decide if it is important to pull a release from PyPi

Having an automatically generated version number avoids the possibility of 7 through 9.

In our projects, we've automated 6 with a bot that monitors the protected main branch. We skip "4" for the release notes PRs. so the time between "agreeing to do a release" to something being published is as low as 10 mins.

Resolving user issues

From a philosophical point of view, I feel that "issues" shouldn't be closed until they resolved on the reported user's machine (Please don't worry about the issues I raise, I'll patch your code if the problem is big enough for me!). So reducing the time to a release is an important step is really getting the feedback you need from an end user who may not be entirely interested in "trying a development release". From an "auditing" standpoint, its also bad for you to encourage them to be on a development release. Today, if somebody (yourself, a collaborator, an intern) gives you the output of the code

import pygfx
print(pygfx.__version__)

you cannot tell if it is version 0.1.18 from pypi or if they are on the main branch where we made some breaking changes (pick_write for example -- thank you!). Personally, I print the output of wgpu.diagnostics.get_report() in some of our debugging code so its great to have this version be highly specific.

My conclusions

Of course none of this trumps a usecase that you all deemed important to yourselves, that is simplicity and compatibility with PyInstaller.

[Korijn]

Thanks for sharing all that, very interesting! Just out of curiosity:

In our projects, we've automated 6 with a bot that monitors the protected main branch.

What is the condition for the bot to step in and apply a tag?

We skip "4" for the release notes PRs.

What is the condition for CI to be skipped? How do you determine a PR is just release notes and not anything else?

Regardless, the following definitely is a recognizable issue we experience in bug reports:

you cannot tell if it is version 0.1.18 from pypi or if they are on the main branch

And this is just plain cool:

start to be "compatible" with version 0.1.19 even before it comes out

It all sounds pretty great. I'm sure we can achieve the benefits without miniver's layers of complexity.

[hmaarrfk]

In our projects, we've automated 6 with a bot that monitors the protected main branch.

Its silly, if something like Merge branch release_.* appears it creates a tag with the appropriate regex.

Its not really open source, so it kinda works for the threat model we have now.

What is the condition for CI to be skipped? How do you determine a PR is just release notes and not anything else?

Your CIs are REALLY fast, I wouldn't do it for this (yet), but ours can take 1-2 hours and that just isn't acceptable for a "daily release". CI time for private projects isn't free, so you have to balance that too. But github actions is just really fast and make much of this "not so important".

Our "release script" is basically:

git checkout main
git pull
git checkout -b "release_${version}"
changes="SOME AUTOMATED WAY TO PULL CHANGES"
git commit -a -m "Release notes for ${version}

${changes}
"
git push --set-upstream origin "release_${version}"

It all sounds pretty great. I'm sure we can achieve the benefits without miniver's layers of complexity.

Yes. Though build infrastructure always takes a bit of working around. So I'm glad I got to understand the landscape of your workflows in my previous PRs. I'll attempt again eventually.

Recently, I've seen many projects have been using for a more streamlined version management
https://pypi.org/project/setuptools-scm/

which may address some of the concerns. I've simply never used it in my own projects.

[almarklein]

Thanks for the detailed explanation @hmaarrfk! The advantages are much more clear to me now.

I think much of the complexity, as well as the problems with portability (e.g. pyinstaller) is that there is not simply a version number in the codebase. If we stick to having that, I think a solution is in close reach.

This does mean we cannot simply push a tag to make a release. But like you said, this is not a big deal for us. And with some of the other tricks you mentioned (e.g. bypass CI for specific commits), there is still room for streamlining.

We could have a _version.py that looks something like this:

# Update this at each release
__version__ = "0.2.0"

from pathlib import Path

is_git_repo = Path(__file__).parents[1].joinpath(".git").is_dir()

if is_git_repo:
    git_version = version_from_git()
    if git_version:
        __version__ = git_version
    else:
        __version__ += ".unknown"

def version_from_git():
    ... probably mostly a copy from miniver