Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark all Python tools as exportable #20787

Merged
merged 5 commits into from
May 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 1 addition & 4 deletions docs/docs/python/overview/lockfiles.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -184,10 +184,7 @@ It is strongly recommended that these tools be installed from a hermetic lockfil

The only time you need to think about this is if you want to customize the tool requirements that Pants uses. This might be the case if you want to modify the version of a tool or add extra requirements (for example, tool plugins).

:::caution Exporting tools requires a custom lockfile
:::

If you want a tool to be installed from some resolve, instead of from the built-in lockfile, you set `install_from_resolve` and `requirements` on the tool's config section:
Tools can also be installed from a specific resolve instead of from the built-in lockfile. This is useful for specifying a version of the tool and including extra packages. To do this, set `install_from_resolve` and `requirements` on the tool's config section:

```toml title="pants.toml"
[python.resolves]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/using-pants/setting-up-an-ide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The `--py-resolve-format=symlinked_immutable_virtualenv` option symlinks to an i

### Tool virtualenvs

`pants export` can also create a virtualenv for each of the Python tools you use via Pants, such as `black`, `isort`, `pytest`, `mypy`, `flake8` and so on. This allows you to configure your editor to use the same version of the tool as Pants does for workflows like formatting on save. Follow [the instructions for creating a tool lockfile](../python/overview/lockfiles#lockfiles-for-tools).
`pants export` can also create a virtualenv for each of the Python tools you use via Pants, such as `black`, `isort`, `pytest`, `mypy`, `flake8` and so on. This allows you to configure your editor to use the same version of the tool as Pants does for workflows like formatting on save. To use a custom version of these tools, follow [the instructions for creating a tool lockfile](../python/overview/lockfiles#lockfiles-for-tools).

## Generated code

Expand Down
2 changes: 2 additions & 0 deletions docs/notes/2.22.x.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ The deprecation for the `platforms` field for the `pex_binary` and `pex_binaries

Python tool subsystem docs and help text now include the default version of the tool, along with instructions on how to override this version using a custom lockfile. Additionally, the help text for the `install_from_resolve` option for Python tools now includes this same information.

Python tools can be [exported from their default bundled lockfiles](https://www.pantsbuild.org/2.22/docs/using-pants/setting-up-an-ide#tool-virtualenvs). For instance, when using the default `black` subsystem, `pants export --resolve=black` will export a venv containing the version of black that Pants runs.

#### Semgrep

The default version of `semgrep` used by the `pants.backends.experimental.tool.semgrep` backend is now version 1.72.0, upgraded from 1.46.0. This version requires Python 3.8 or greater.
Expand Down
6 changes: 5 additions & 1 deletion src/python/pants/backend/cc/lint/clangformat/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import Rule, collect_rules
from pants.engine.unions import UnionRule
Expand Down Expand Up @@ -51,4 +52,7 @@ def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:


def rules() -> Iterable[Rule | UnionRule]:
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, ClangFormat),
]
2 changes: 2 additions & 0 deletions src/python/pants/backend/python/goals/coverage_py.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
PythonSourceFiles,
PythonSourceFilesRequest,
)
from pants.core.goals.resolves import ExportableTool
from pants.core.goals.test import (
ConsoleCoverageReport,
CoverageData,
Expand Down Expand Up @@ -621,4 +622,5 @@ def rules():
return [
*collect_rules(),
UnionRule(CoverageDataCollection, PytestCoverageDataCollection),
UnionRule(ExportableTool, CoverageSubsystem),
]
5 changes: 0 additions & 5 deletions src/python/pants/backend/python/goals/export_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,10 @@
from pants.backend.python.util_rules import local_dists_pep660, pex_from_targets
from pants.base.specs import RawSpecs
from pants.core.goals.export import ExportResults
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules import distdir
from pants.engine.internals.parametrize import Parametrize
from pants.engine.rules import QueryRule
from pants.engine.target import Targets
from pants.engine.unions import UnionRule
from pants.testutil.rule_runner import RuleRunner
from pants.util.frozendict import FrozenDict

Expand All @@ -48,9 +46,6 @@ def rule_runner() -> RuleRunner:
*distdir.rules(),
*local_dists_pep660.rules(),
*isort_subsystem.rules(), # add a tool that we can try exporting
UnionRule(
ExportableTool, isort_subsystem.Isort
), # TODO: remove this manual export when we add ExportableTool to tools
QueryRule(Targets, [RawSpecs]),
QueryRule(ExportResults, [ExportVenvsRequest]),
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, SkipOption


Expand All @@ -29,4 +31,7 @@ class AddTrailingComma(PythonToolBase):


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, AddTrailingComma),
]
7 changes: 6 additions & 1 deletion src/python/pants/backend/python/lint/autoflake/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, SkipOption


Expand All @@ -32,4 +34,7 @@ class Autoflake(PythonToolBase):


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, Autoflake),
]
3 changes: 3 additions & 0 deletions src/python/pants/backend/python/lint/bandit/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,11 @@
InterpreterConstraintsField,
PythonSourceField,
)
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import collect_rules
from pants.engine.target import FieldSet, Target
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, FileOption, SkipOption


Expand Down Expand Up @@ -72,4 +74,5 @@ def rules():
return (
*collect_rules(),
*lockfile.rules(),
UnionRule(ExportableTool, Bandit),
)
7 changes: 6 additions & 1 deletion src/python/pants/backend/python/lint/black/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@
InterpreterConstraintsField,
PythonSourceField,
)
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import collect_rules
from pants.engine.target import FieldSet, Target
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, BoolOption, FileOption, SkipOption
from pants.util.strutil import softwrap

Expand Down Expand Up @@ -89,4 +91,7 @@ def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, Black),
]
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, SkipOption


Expand All @@ -25,4 +27,7 @@ class Docformatter(PythonToolBase):


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, Docformatter),
]
5 changes: 3 additions & 2 deletions src/python/pants/backend/python/lint/flake8/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@

from dataclasses import dataclass

from pants.backend.python.goals import lockfile
from pants.backend.python.lint.flake8.skip_field import SkipFlake8Field
from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import (
Expand All @@ -20,12 +19,14 @@
PythonSourceFilesRequest,
StrippedPythonSourceFiles,
)
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.addresses import Addresses, UnparsedAddressInputs
from pants.engine.fs import AddPrefix, Digest
from pants.engine.internals.native_engine import EMPTY_DIGEST
from pants.engine.rules import Get, collect_rules, rule
from pants.engine.target import FieldSet, Target, TransitiveTargets, TransitiveTargetsRequest
from pants.engine.unions import UnionRule
from pants.option.option_types import (
ArgsListOption,
BoolOption,
Expand Down Expand Up @@ -211,6 +212,6 @@ async def flake8_first_party_plugins(flake8: Flake8) -> Flake8FirstPartyPlugins:
def rules():
return (
*collect_rules(),
*lockfile.rules(),
*python_sources.rules(),
UnionRule(ExportableTool, Flake8),
)
3 changes: 3 additions & 0 deletions src/python/pants/backend/python/lint/isort/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, BoolOption, FileListOption, SkipOption
from pants.util.strutil import softwrap

Expand Down Expand Up @@ -93,4 +95,5 @@ def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:
def rules():
return [
*collect_rules(),
UnionRule(ExportableTool, Isort),
]
3 changes: 3 additions & 0 deletions src/python/pants/backend/python/lint/pydocstyle/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,11 @@
from pants.backend.python.lint.pydocstyle.skip_field import SkipPydocstyleField
from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript, PythonSourceField
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import collect_rules
from pants.engine.target import FieldSet, Target
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, BoolOption, FileOption, SkipOption
from pants.util.docutil import bin_name
from pants.util.strutil import softwrap
Expand Down Expand Up @@ -91,4 +93,5 @@ def rules():
return (
*collect_rules(),
*lockfile.rules(),
UnionRule(ExportableTool, Pydocstyle),
)
5 changes: 3 additions & 2 deletions src/python/pants/backend/python/lint/pylint/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
from dataclasses import dataclass
from typing import Iterable

from pants.backend.python.goals import lockfile
from pants.backend.python.lint.pylint.skip_field import SkipPylintField
from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import (
Expand All @@ -22,11 +21,13 @@
PythonSourceFilesRequest,
StrippedPythonSourceFiles,
)
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.addresses import Addresses, UnparsedAddressInputs
from pants.engine.fs import EMPTY_DIGEST, AddPrefix, Digest
from pants.engine.rules import Get, collect_rules, rule
from pants.engine.target import FieldSet, Target, TransitiveTargets, TransitiveTargetsRequest
from pants.engine.unions import UnionRule
from pants.option.option_types import (
ArgsListOption,
BoolOption,
Expand Down Expand Up @@ -202,5 +203,5 @@ async def pylint_first_party_plugins(pylint: Pylint) -> PylintFirstPartyPlugins:
def rules():
return (
*collect_rules(),
*lockfile.rules(),
UnionRule(ExportableTool, Pylint),
)
7 changes: 6 additions & 1 deletion src/python/pants/backend/python/lint/pyupgrade/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, SkipOption


Expand All @@ -28,4 +30,7 @@ class PyUpgrade(PythonToolBase):


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, PyUpgrade),
]
3 changes: 3 additions & 0 deletions src/python/pants/backend/python/lint/ruff/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@
from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.backend.python.util_rules import python_sources
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, BoolOption, FileOption, SkipOption
from pants.util.strutil import softwrap

Expand Down Expand Up @@ -86,4 +88,5 @@ def rules():
return (
*collect_rules(),
*python_sources.rules(),
UnionRule(ExportableTool, Ruff),
)
7 changes: 6 additions & 1 deletion src/python/pants/backend/python/lint/yapf/subsystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, BoolOption, FileOption, SkipOption
from pants.util.strutil import softwrap

Expand Down Expand Up @@ -88,4 +90,7 @@ def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, Yapf),
]
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@

from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption
from pants.util.strutil import help_text

Expand Down Expand Up @@ -32,4 +34,7 @@ class PyOxidizer(PythonToolBase):


def rules():
return collect_rules()
return [
*collect_rules(),
UnionRule(ExportableTool, PyOxidizer),
]
5 changes: 3 additions & 2 deletions src/python/pants/backend/python/subsystems/ipython.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,11 @@

from __future__ import annotations

from pants.backend.python.goals import lockfile
from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import ConsoleScript
from pants.core.goals.resolves import ExportableTool
from pants.engine.rules import collect_rules
from pants.engine.unions import UnionRule
from pants.option.option_types import BoolOption
from pants.util.strutil import softwrap

Expand Down Expand Up @@ -41,5 +42,5 @@ class IPython(PythonToolBase):
def rules():
return (
*collect_rules(),
*lockfile.rules(),
UnionRule(ExportableTool, IPython),
)
5 changes: 3 additions & 2 deletions src/python/pants/backend/python/subsystems/pytest.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
from dataclasses import dataclass
from typing import Iterable

from pants.backend.python.goals import lockfile
from pants.backend.python.subsystems.python_tool_base import PythonToolBase
from pants.backend.python.target_types import (
ConsoleScript,
Expand All @@ -20,11 +19,13 @@
PythonTestsXdistConcurrencyField,
SkipPythonTestsField,
)
from pants.core.goals.resolves import ExportableTool
from pants.core.goals.test import RuntimePackageDependenciesField, TestFieldSet
from pants.core.util_rules.config_files import ConfigFilesRequest
from pants.core.util_rules.environments import EnvironmentField
from pants.engine.rules import collect_rules
from pants.engine.target import Target
from pants.engine.unions import UnionRule
from pants.option.option_types import ArgsListOption, BoolOption, FileOption, SkipOption, StrOption
from pants.util.strutil import softwrap

Expand Down Expand Up @@ -155,5 +156,5 @@ def config_request(self, dirs: Iterable[str]) -> ConfigFilesRequest:
def rules():
return (
*collect_rules(),
*lockfile.rules(),
UnionRule(ExportableTool, PyTest),
)