Use `vcs_version` for Pants' version #20438

thejcannon · 2024-01-18T18:58:12Z

This change makes it so we can version a release simply from tagging a commit (therefore not requiring a PR to release)

It accomplishes this by:

Removing the VERSION file
- We'll likely want to update scie-pants to reflect this change
Updates version.py to use setuptools_scm to generate the version on-the-fly if RUNNING_PANTS_FROM_SOURCES
Adds version_scheme and local_scheme fields to vcs_version and wires them up
- This is because setuptools_scm doesn't support .devN builds that aren't .dev0 without a different scheme
Changing the packaged VERSION file (at src/python/pants/_version/VERSION) to be generated via vcs_version
Adds a pants_dev_requirements.txt file which is solely used for Pants dev venv. And add setuptools_scm
Updating the docs/automation as well

Demo:

# Using pants or ./pants
josh@cephandrius:~/work/pants$ pants version
2.20.0.dev6+geebeb404a8.d20240118
josh@cephandrius:~/work/pants$ ./pants version
2.20.0.dev6+geebeb404a8.d20240118

# With/without daemon
josh@cephandrius:~/work/pants$ pants --no-pantsd version
Pantsd has been turned off via Flag.
2.20.0.dev6+g64b67dc7ea
josh@cephandrius:~/work/pants$ ./pants --no-pantsd version
Pantsd has been turned off via Flag.
2.20.0.dev6+g64b67dc7ea

# Building the real-deal, locally
josh@cephandrius:~/work/pants$ ./pants package src/python/pants:pants-pex && cd $(mktemp -d) && touch pants.toml && NO_SCIE_WARNING=1 ~/work/pants/dist/src.python.pants/pants-pex.pex
 version
...
2.20.1.dev13+g64b67dc7ea

# And in CI (note I have to tag this commit)
josh@cephandrius:~/work/pants$ git tag release_2.20.0.dev7
josh@cephandrius:~/work/pants$ CI=1 ./pants package src/python/pants:pants-pex && cd $(mktemp -d) && touch pants.toml && NO_SCIE_WARNING=1 ~/work/pants/dist/src.python.pants/pants-pex.pex version
...
2.20.0.dev7

# And lastly, multiple tags (e.g. rc and stable)
josh@cephandrius:~/work/pants$ git tag release_2.20.0rc1
josh@cephandrius:~/work/pants$ git tag release_2.20.0
josh@cephandrius:~/work/pants$ CI=1 ./pants package src/python/pants:pants-pex && cd $(mktemp -d) && touch pants.toml && NO_SCIE_WARNING=1 ~/work/pants/dist/src.python.pants/pants-pex.pex version
...
2.20.0

thejcannon · 2024-01-18T18:58:29Z

Draft PR until I fix the Rust side of things and actually update the docs/automation

benjyw · 2024-01-18T22:13:42Z

I love the idea! So when do wheels get built?

I guess we always build wheels in main and release branches anyway?

benjyw

LGTM mod the rust bit of course!

src/python/pants/bin/pants_runner.py

thejcannon · 2024-01-18T23:16:47Z

I guess we always build wheels in main and release branches anyway?

Yup!

huonw · 2024-01-19T00:22:40Z

Nice! I like the idea.

The version looks quite "rich", potentially changing for every single commit. I wonder if this will cause cache hit rates for tests to be essentially zero: anything that depends on the version will have its digest change every build, and, I imagine most tests do (transitively) via things like doc_url? (I'm on mobile so can't confirm)

If this is the case, can we reduce the impact somehow?

thejcannon · 2024-01-19T02:03:59Z

Nice! I like the idea.

The version looks quite "rich", potentially changing for every single commit. I wonder if this will cause cache hit rates for tests to be essentially zero: anything that depends on the version will have its digest change every build, and, I imagine most tests do (transitively) via things like doc_url? (I'm on mobile so can't confirm)

If this is the case, can we reduce the impact somehow?

Welp, I was ready to tell you the multiple reasons it won't be a problem, but that just led to a different, worse problem. All the tests just die 😂

So in fixing the deaths, I'll make sure we just use some constant version for tests (like 3.0)

thejcannon · 2024-01-19T02:09:59Z

Ok @huonw I moved the VERSION dep to the wheel, and then set _PANTS_VERSION_OVERRIDE to 3.0.0 in pants.toml for tests. 🎉

kaos

cool stuff

pants.toml

3rdparty/python/pants_dev_requirements.txt

src/rust/engine/pantsd/build.rs

benjyw

Preferably the new fields on vcs_version would be in a separate commit from the changes in the Pants repo, so it's obvious that there are two parts here - a user facing feature enhancement, and an internal Pants repo application of that.

src/rust/engine/pantsd/src/pantsd_testing.rs

docs/docs/contributions/releases/release-process.mdx

thejcannon · 2024-01-22T13:24:26Z

Preferably the new fields on vcs_version would be in a separate commit from the changes in the Pants repo, so it's obvious that there are two parts here - a user facing feature enhancement, and an internal Pants repo application of that.

#20446

thejcannon · 2024-01-22T13:57:17Z

Hmm the failure alludes me.

The fatal: No tags can describe '4c62b40a103d0014fd3277059efc1333211141f9'. makes sense. We maybe want some kind of "fallback" in the case of very shallow clones like checkout Action uses.

But the ModuleNotFoundError: No module named 'setuptools_scm' doesn't make sense. "Successfully installed debugpy-1.6.0 pydevd-pycharm-203.5419.8 setuptools_scm-8.0.4" from here

kaos · 2024-01-22T16:02:33Z

Have we deprecated the use of PANTS_SHA now then?
As that will stop to work once we remove src/python/pants/VERSION.

https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

thejcannon · 2024-01-22T19:12:16Z

Have we deprecated the use of PANTS_SHA now then?
As that will stop to work once we remove src/python/pants/VERSION.

https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

Yes, IIRC we deprecated it a while back with the suggestion of just checking out the repo locally and using PANTS_FROM_SOURCES

kaos · 2024-01-23T08:45:23Z

Have we deprecated the use of PANTS_SHA now then?
As that will stop to work once we remove src/python/pants/VERSION.
https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

Yes, IIRC we deprecated it a while back with the suggestion of just checking out the repo locally and using PANTS_FROM_SOURCES

Excellent. That'll make life a little easier over in scie-pants land.

thejcannon · 2024-01-23T14:09:36Z

Found it: https://github.com/pantsbuild/scie-pants/blob/main/CHANGES.md#L43

kaos · 2024-01-24T09:02:59Z

Found it: https://github.com/pantsbuild/scie-pants/blob/main/CHANGES.md#L43

Ah yea, me also: https://github.com/pantsbuild/scie-pants/pull/351/files#diff-d975bf659606195d2165918f93e1cf680ef68ea3c9cab994f033705fea8238b2 :)

src/rust/engine/pantsd/src/pantsd_testing.rs

docs/docs/contributions/releases/release-process.mdx

huonw · 2024-02-07T02:55:55Z

src/python/pants/_version/BUILD

+    # NB: Don't dirty the version in CI
+    local_scheme="no-local-version" if env("CI", "0") == "1" else None,


WDYT about doing this unconditionally? Why might it be handy?

Suggested change

# NB: Don't dirty the version in CI

local_scheme="no-local-version" if env("CI", "0") == "1" else None,

local_scheme="no-local-version",

User reports, mostly. Whens someone is using PANTS_SOURCE=....

src/python/pants/_version/BUILD

pants.toml

huonw · 2024-02-07T03:07:29Z

src/python/pants/version.py

 VERSION: str = (
    # Do not remove/change this env var without coordinating with `pantsbuild/scie-pants` as it is
    # being used when bootstrapping Pants with a released version.
    os.environ.get(_PANTS_VERSION_OVERRIDE)
    or
+    # NB: This is only relevant for the Pants repo itself
+    (
+        _determine_version_from_pants_source()


Do we need to computing the version 'accurately' when running from sources? What's the downside if we just substituted in a fixed dummy version here, and thus avoided needing to pull in setuptools_scm?

In particular, I wonder if Pants supports running from a git-less tarball currently... if so, requiring setuptools_scm and its git interactions might break it?

In those cases (which we don't document supporting currently) you're outside the normal flow. in that case, you'd just set _PANTS_VERSION_OVERRIDE on invocation (among other hacks, I imagine)

Oh and I guess to answer your questions. I'd rather err on the side of caution. Where we could reasonably ask users to run --version and it gives us something useful.

Then, if users want to exist outside of this model, they can do so by either setting the env var, or modifying the file.

I'm also toying with speeding up option registration by doing it at build time instead of runtime, and in that case it would have to be tied to a specific version, and it would be chaotic to not invalidate that correctly.

Just so I'm clear, is that something that'll preclude and block this PR? Or is it something that'll be handled in yours? Or something else?

Just trying to understand the impact of your statement.

I'm just providing a reason that every version should have a well-defined version, even when running from sources, rather than using a fixed dummy.

src/python/pants/BUILD

Co-authored-by: Huon Wilson <huon@exoflare.io>

kaos · 2024-02-14T14:39:05Z

Have we deprecated the use of PANTS_SHA now then?
As that will stop to work once we remove src/python/pants/VERSION.
https://github.com/pantsbuild/scie-pants/blob/2cd4278eb8bf773b398b88cdf6ef64fe5a4843bf/tools/src/scie_pants/pants_version.py#L153-L155

Yes, IIRC we deprecated it a while back with the suggestion of just checking out the repo locally and using PANTS_FROM_SOURCES

Ah! But PANTS_SOURCE feature also relies on the VERSION file:

https://github.com/pantsbuild/scie-pants/blob/c213edb28626973d6262c5016ad1148e35f08526/src/main.rs#L284-L290

So, that at least would need to be fixed along with this change, then I think..

huonw

Getting closer!

Good catch re the scie-pants checks. Even if we do land the scie-pants change, landing this might be a breaking change, as anyone using PANTS_SOURCE will start getting errors that are very opaque:

Error: No such file or directory (os error 2)

Given PANTS_SOURCE is a fairly niche/experts-only feature, that's probably okay? So, instead of doing a deprecation cycle with

pants/src/python/pants/bin/pants_runner.py

Line 27 in 5779896

MINIMUM_SCIE_PANTS_VERSION = Version("0.10.0")

we just do a hard-break and communicate with users via release notes/other announcements? Either that, or do do a deprecation cycle flagging the problem if running with PANTS_SOURCE. Thoughts?

huonw · 2024-02-14T00:12:11Z

src/python/pants_release/generate_github_workflows.py

@@ -170,7 +170,7 @@ def ensure_category_label() -> Sequence[Step]:

 def checkout(
    *,
-    fetch_depth: int = 10,
+    fetch_depth: int = 250,


Why does this change?

huonw · 2024-02-19T00:09:59Z

cargo

+GIT_TAG=$(git describe --tags)
+export GIT_TAG


What do you think about giving this a more distinct name? The current name is generic enough that it could come from a CI provider or similar? E.g. PANTS_CARGO_GIT_TAG or something (although the PANTS_ prefix is potentially something to avoid)?

Also, does this need to sync up with the version used in Python? If so, this might be different when running from source?

kaos · 2024-02-19T07:45:38Z

Getting closer!

Good catch re the scie-pants checks. Even if we do land the scie-pants change, landing this might be a breaking change, as anyone using PANTS_SOURCE will start getting errors that are very opaque:
Error: No such file or directory (os error 2)
Given PANTS_SOURCE is a fairly niche/experts-only feature, that's probably okay? So, instead of doing a deprecation cycle with

pants/src/python/pants/bin/pants_runner.py

Line 27 in 5779896

MINIMUM_SCIE_PANTS_VERSION = Version("0.10.0")

we just do a hard-break and communicate with users via release notes/other announcements? Either that, or do do a deprecation cycle flagging the problem if running with PANTS_SOURCE. Thoughts?

Yea, I think if we release a new version of scie-pants that handles this properly, the first thing we ask whenever users face issues like above is which version of scie-pants they're using and ask to upgrade to latest first, which would help in this case. Being a dev/debug feature and not a "released" pants feature per se, I'm OK with not providing a deprecation cycle to change this behavior given an easy out with just upgrading scie-pants.

huonw · 2024-02-27T02:57:49Z

Another question: if the version is only known once tagged, we can easily forget to handle deprecations (e.g. remove or postpone) before tagging, and it seems they can happen in a way that causes the release build to break. Is that something we should be prepared to handle?

In particular, #20609 sets up 2.21.0.dev0, but CI fails hard when bootstrap pants, with errors like the following:

pants.base.deprecated.CodeRemovedError: DEPRECATED: option 'remote_oauth_bearer_token_path' in global scope is scheduled to be removed in version 2.21.0.dev0.

Referring to

pants/src/python/pants/option/global_options.py

Lines 1585 to 1601 in 8481389

    
               remote_oauth_bearer_token_path = StrOption( 
        
                   default=None, 
        
                   advanced=True, 
        
                   help=softwrap( 
        
                       """ 
        
                       Path to a file containing an oauth token to use for gGRPC connections to 
        
                       `[GLOBAL].remote_execution_address` and `[GLOBAL].remote_store_address`. 
        
                       If specified, Pants will add a header in the format `authorization: Bearer <token>`. 
        
                       You can also manually add this header via `[GLOBAL].remote_execution_headers` and 
        
                       `[GLOBAL].remote_store_headers`, or use `[GLOBAL].remote_auth_plugin` to provide a plugin to 
        
                       dynamically set the relevant headers. Otherwise, no authorization will be performed. 
        
                       """ 
        
                   ), 
        
                   removal_version="2.21.0.dev0", 
        
                   removal_hint=f'use `[GLOBAL].remote_oauth_bearer_token = "@/path/to/token.txt"` instead, see {doc_url("reference/global-options#remote_oauth_bearer_token")}', 
        
               )

(NB. I'm a bit confused by this, because the corresponding deprecation warning doesn't appear on normal builds. 🤷‍♂️ )

In this case, it was caught by PR CI, but if I had just tagged 2.21.0.dev0 directly to trigger the release build, we might end up with a situation where we tag a release and it either completely fails to build, or is very non-functional when shipped.

In either case, I imagine we can usually just fix the problem and do the next release with low impact... assuming we get the deprecations right so that it's always dev release where behaviour changes, and not a stable release.

But, we don't always get it right (see `crossversion="partial" in #20616). So, should we think of ways to mitigate this risk? Just brainstorming ideas:

some sort of pre-tagging testing
dynamic or static linting about how the "highest risk" fields like warn_or_error and deprecated_version are used, to check that they're always referring to .dev0 releases?
something else?

kaos · 2024-02-27T08:48:47Z

dynamic or static linting about how the "highest risk" fields like warn_or_error and deprecated_version are used, to check that they're always referring to .dev0 releases?

This makes sense to me.

if the version is only known once tagged, we can easily forget to handle deprecations

I think one way to smoke this out is that you can override the version locally when running tests... but then running the full suite is likely not practical so wouldn't catch everything.

cognifloyd · 2024-04-15T17:09:13Z

Maybe we need a new goal that lists upcoming deprecations?

thejcannon added 2 commits January 18, 2024 12:37

woah it works

64b67dc

version_scheme

29e086f

thejcannon added the category:internal CI, fixes for not-yet-released features, etc. label Jan 18, 2024

thejcannon added this to the 2.19.x milestone Jan 18, 2024

thejcannon requested review from stuhood, kaos, benjyw and huonw January 18, 2024 18:58

benjyw reviewed Jan 18, 2024

View reviewed changes

src/python/pants/bin/pants_runner.py Show resolved Hide resolved

make sure tests don't die

b7c7c23

kaos reviewed Jan 19, 2024

View reviewed changes

pants.toml Outdated Show resolved Hide resolved

3rdparty/python/pants_dev_requirements.txt Outdated Show resolved Hide resolved

thejcannon added 2 commits January 19, 2024 11:31

Add comment

b3c1b16

embed the tag

7a2ad13

thejcannon commented Jan 19, 2024

View reviewed changes

src/rust/engine/pantsd/build.rs Outdated Show resolved Hide resolved

thejcannon added 3 commits January 19, 2024 13:06

fmt

ccef094

Merge branch 'main' into jcannon/vcs-version

db28c1c

docs and process

d76b253

thejcannon marked this pull request as ready for review January 19, 2024 20:51

thejcannon added 5 commits January 19, 2024 15:25

no amp

e8fb280

try export

a0ca6ab

thatll do it

39fb59e

try this

e8f887c

unused

4b5ce76

kaos approved these changes Jan 20, 2024

View reviewed changes

use 3.0 in rust tests as well

e1ed9af

benjyw reviewed Jan 21, 2024

View reviewed changes

src/rust/engine/pantsd/src/pantsd_testing.rs Outdated Show resolved Hide resolved

docs/docs/contributions/releases/release-process.mdx Show resolved Hide resolved

benjyw approved these changes Jan 21, 2024

View reviewed changes

thejcannon mentioned this pull request Jan 22, 2024

Add version/local scheme fields to vcs_version #20446

Merged

0.0.0.dev0

7bad224

benjyw removed this from the 2.19.x milestone Jan 22, 2024

thejcannon commented Feb 3, 2024

View reviewed changes

src/rust/engine/pantsd/src/pantsd_testing.rs Outdated Show resolved Hide resolved

Update src/rust/engine/pantsd/src/pantsd_testing.rs

397b016

huonw reviewed Feb 7, 2024

View reviewed changes

thejcannon and others added 7 commits February 7, 2024 10:15

Apply suggestions from code review

b77b227

Co-authored-by: Huon Wilson <huon@exoflare.io>

Merge branch 'main' into jcannon/vcs-version

6cee103

always None

dbff533

some comments

1148177

bump depth

f6b37d9

fix the version

bcbef9c

use only-version

2676f8e

huonw reviewed Feb 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use `vcs_version` for Pants' version #20438

Use `vcs_version` for Pants' version #20438

thejcannon commented Jan 18, 2024

thejcannon commented Jan 18, 2024

benjyw commented Jan 18, 2024 •

edited

Loading

benjyw left a comment

thejcannon commented Jan 18, 2024

huonw commented Jan 19, 2024

thejcannon commented Jan 19, 2024

thejcannon commented Jan 19, 2024

kaos left a comment

benjyw left a comment

thejcannon commented Jan 22, 2024

thejcannon commented Jan 22, 2024

kaos commented Jan 22, 2024

thejcannon commented Jan 22, 2024

kaos commented Jan 23, 2024

thejcannon commented Jan 23, 2024 •

edited

Loading

kaos commented Jan 24, 2024

huonw Feb 7, 2024

thejcannon Feb 7, 2024

huonw Feb 7, 2024

thejcannon Feb 7, 2024

thejcannon Feb 12, 2024

benjyw Feb 16, 2024

thejcannon Feb 16, 2024

benjyw Feb 17, 2024

kaos commented Feb 14, 2024

huonw left a comment

huonw Feb 14, 2024

huonw Feb 19, 2024

kaos commented Feb 19, 2024

huonw commented Feb 27, 2024

kaos commented Feb 27, 2024

cognifloyd commented Apr 15, 2024

		# NB: Don't dirty the version in CI
		local_scheme="no-local-version" if env("CI", "0") == "1" else None,

	# NB: Don't dirty the version in CI
	local_scheme="no-local-version" if env("CI", "0") == "1" else None,
	local_scheme="no-local-version",

Use vcs_version for Pants' version #20438

Are you sure you want to change the base?

Use vcs_version for Pants' version #20438

Conversation

thejcannon commented Jan 18, 2024

thejcannon commented Jan 18, 2024

benjyw commented Jan 18, 2024 • edited Loading

benjyw left a comment

Choose a reason for hiding this comment

thejcannon commented Jan 18, 2024

huonw commented Jan 19, 2024

thejcannon commented Jan 19, 2024

thejcannon commented Jan 19, 2024

kaos left a comment

Choose a reason for hiding this comment

benjyw left a comment

Choose a reason for hiding this comment

thejcannon commented Jan 22, 2024

thejcannon commented Jan 22, 2024

kaos commented Jan 22, 2024

thejcannon commented Jan 22, 2024

kaos commented Jan 23, 2024

thejcannon commented Jan 23, 2024 • edited Loading

kaos commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaos commented Feb 14, 2024

huonw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaos commented Feb 19, 2024

huonw commented Feb 27, 2024

kaos commented Feb 27, 2024

cognifloyd commented Apr 15, 2024

Use `vcs_version` for Pants' version #20438

Use `vcs_version` for Pants' version #20438

benjyw commented Jan 18, 2024 •

edited

Loading

thejcannon commented Jan 23, 2024 •

edited

Loading