Configure pants to look for plugins in pants-plugins/ directory

[cognifloyd]

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

• pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
• pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

This configures pants and generates a lockfile for a new pants-plugins/ directory.

In follow-up PRs I will add several plugins for pants to improve Developer Experience and simplify config we need to add in a lot of directories. This PR adds the prerequisite global config we need before introducing any of those plugins.

This adjusts config in two files: pants.toml and pants-plugins/BUILD.

Relevant Pants documentation

• Writing Plugins | Plugins overview
  • in repo plugins
  • Building in-repo plugins with Pants
• Targets:
  • pants_requirements

pants plugins

The pants plugin framework is very flexible. The core engine of pants is written in rust (for speed), but the plugin API is python. Each of the backends we have enabled, black, flake8, etc, are implemented as pants plugins. Plus, we can (and will) add in-repo plugins to handle some of the unique aspects of how StackStorm is organized.

By convention in-repo plugins go in the pants-plugins/ directory.

pants-plugins have a dependency on pants itself

At risk of stating the obvious, all of our in-repo pants-plugins (once added, will) depend on pants. We could add a python_requirement target to capture that dependency, but then updating pants would require updating the same version number in 2 places. Luckily pants provides some tools to help here.

In pants.toml, we register the pants.backend.plugin_development backend to get access to the pants_requirements target.

st2/pants.toml

Line 25 in 87af527

"pants.backend.plugin_development",

And then use pants_requirements in pants-plugins/BuILD:

st2/pants-plugins/BUILD

Lines 8 to 12 in 87af527

	# this adds a dependency on the pants libs using the version specified in pants.toml
	pants_requirements(
	name="pants",
	testutil=False,
	)

Configure pants to find plugins in pants-plugins/

First we add pants-plugins to [GLOBAL].pythonpath. This pythonpath only refers to the path pants uses when loading itself, not our code or the other tools we use.

st2/pants.toml

Line 10 in 87af527

pythonpath = ["%(buildroot)s/pants-plugins"]

And we register another source root, so we can run black, flake8 and friends on our pants-plugins/ code:

st2/pants.toml

Lines 84 to 85 in 87af527

	# pants plugins
	"/pants-plugins",

pants-plugins resolve and lockfile

When we add an in-repo plugins we should keep this warning from the docs in mind:

In-repo dependencies
In-repo plugin code should not depend on other in-repo code outside of the pants-plugins folder. The pants-plugins folder helps isolate plugins from regular code, which is necessary due to how Pants's startup sequence works.

To better isolate the pants-plugins code from the rest of our code, we create a new resolve for it.

This registers the resolve+lockfile in pants.toml:

st2/pants.toml

Line 105 in 87af527

pants-plugins = "lockfiles/pants-plugins.lock"

And then this adds python interpreter constraints to that resolve. These constraints are for code that runs in pants, so we match the python versions that pants itself uses.

st2/pants.toml

Lines 107 to 113 in 87af527

	[python.resolves_to_interpreter_constraints]
	pants-plugins = [
	# this should match the pants interpreter_constraints:
	# https://github.com/pantsbuild/pants/blob/2.14.x/pants.toml#L125
	# See: https://www.pantsbuild.org/docs/prerequisites
	"CPython>=3.7,<3.10",
	]

And we use __defaults__ to make sure all of the plugin code, including our pants_requirements target, are part of the pants-plugins resolve. aside: I also added skip_pylint=True like in #5837. Because we're using __defaults__ we do not have to merge #5837 before this PR. The setting will apply as soon as it is available.

st2/pants-plugins/BUILD

Lines 1 to 6 in 87af527

	__defaults__(
	all=dict(
	resolve="pants-plugins",
	skip_pylint=True,
	)
	)

And finally, generate lockfiles/pants-plugins.lock using ./pants generate-lockfiles --resolve=pants-plugins.

pants-plugins/README.md

I also added a skeleton readme to explain what goes in the pants-plugins/ directory.

[cognifloyd]

I set testutil=False (defaults to True), because I haven't used it so far. We can add it once we introduce some test(s) that use it.

see: https://www.pantsbuild.org/docs/reference-pants_requirements#codetestutilcode

[amanda11]

LGTM - and thanks for the comments like one on pants-plugins so we know how we have to put to the same version as pants interpreter.