[Python] Wrong SCM version detection both in setup.py and crossbow

[asfimport]

Wanted to install nightly wheels for a rust integration test when I noticed that the nightly wheels get produced with base version 3.0.0.
This is due to the recently changed release workflow. Now we're creating tags from release branches without a post rebase step, so setuptools_scm fails to find the correct version number on diverged branches.

Reporter: Krisztian Szucs / @kszucs
Assignee: Krisztian Szucs / @kszucs

_{Note: This issue was originally created as ARROW-13083. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
We cannot use the git describe command with this workflow since it calculates the distance since the last tag reachable from the current commit.

An alternative could be to manually calculate the distance from the merge-base of the current commit and the latest tag where the latest should be defined by setup.py::default_version variable which we periodically bump after each release.

[asfimport]

Krisztian Szucs / @kszucs:
setuptools_scm's API only allows us to pass a git describe command, but git describe won't reach the new tags from the master branch since they are placed on release branches which diverge from master. We could calculate a version number ourselves but vendoring internal code from setuptools_scm is questionable.

Another approach could be to merge (not rebase) the release branches to the master branch so git describe can reach the tags.

cc @pitrou

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
What we do in pandas is directly after branching adding a tag on master with "next.dev" version. So eg when branching 3.x, we tag master with 4.0.0.dev. The versions based on git describe should work fine.

[asfimport]

Krisztian Szucs / @kszucs:
Thanks for the suggestion @jorisvandenbossche, that's a nice workaround! The question is, shall we tag the fork point or the commit where we bump the development versions? There are intermediate commits.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Ah, yes, we already have a commit to update to the next version. I think it would be logical to tag that commit (assuming it is in theory the first one after branching?)

[asfimport]

Krisztian Szucs / @kszucs:
It isn't the immediate commit after the branching point. If we want to be pedantic than we should tag the first commit right after the branching point as apache-arrow-bumped-version.dev and don't tie it to the version bump commit.

cc @kou

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
But if we want to be pedantic, then the version bump commit should be the first commit after the branching point ;-)

(so I would personally not be too pedantic and tie the tag to the version bump commit, which seems most logical to me, and just try to do it not too far after the branching point)

[asfimport]

Krisztian Szucs / @kszucs:

But if we want to be pedantic, then the version bump commit should be the first commit after the branching point

Ideally yes, though we usually create multiple release candidates. We would need to first create a commit on master, then create the release branch from the preceding commit for the first RC. For every subsequent RCs we would need to cherry-pick commits (potentially required patches) from master and regenerate the changelog again to the release branch, then merge that changelog back to the master :)

It's more logical to me as well to tag the version bump commit, though the in-between commits will have wrong versions generated (which we could live with IMO).

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Ah, yes, I see. I would also say that we can live with that.

[asfimport]

Krisztian Szucs / @kszucs:
This solution sounds good to me. I'm going to create the required tag if @kou agrees.

[asfimport]

Kouhei Sutou / @kou:
I agree with this solution!

[asfimport]

Krisztian Szucs / @kszucs:
Tested this out locally, the version is properly generated after placing apache-arrow-5.0.0.dev tag on master.
Pushed the tag upstream: https://github.com/apache/arrow/releases/tag/apache-arrow-5.0.0.dev
Also updated the release management guide to create a tag after the version bumps: https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-Post-releasetasks