Help api types #14081

kaos · 2022-01-05T15:41:44Z

Builds on top of #13993
Add help info for all available rule output types, and the rules that provides them.

Example output:

$ ./pants help api-types

Plugin API Types
----------------

AccessTokenAcquisition                       

AccessTokenAcquisitionGoalOptions            

AddressToDependees                           AddressToDependees(mapping: pants.util.frozendict.FrozenDict[pants.build_graph.address.Address, pants.util.ordered_set.FrozenOrderedSet[pants.build_graph.address.Address]])

AllGoTargets                                 

AllJavaTargets                               

AllJvmArtifactTargets                        

AllJvmTypeProvidingTargets                   

AllOwnedSources                              All files in the project already owned by targets.

AllProtobufTargets                           

AllPytestPluginSetups                        

AllPythonTargets                             AllPythonTargets(first_party: 'tuple[Target, ...]', third_party: 'tuple[Target, ...]')

AllScalaTargets                              

AllShellTargets                              

AllSourceRoots                               

AllThirdPartyPackages                        All the packages downloaded from a go.mod, along with a digest of the downloaded files.  The digest has files in the format `gopath/pkg/mod`, which is what `GoSdkProcess` sets `GOPATH` to. This means that you
                                             can include the digest in a process and Go will properly consume it as the `GOPATH`.

AllThriftTargets                             

AncestorFiles                                Any ancestor files found.

[...]
ValidateSubsystem                            

VenvPex                                      VenvPex(digest: 'Digest', pex_filename: 'str', pex: 'Script', python: 'Script', bin: 'FrozenDict[str, Script]', venv_rel_dir: 'str')

VenvPexRequest                               VenvPexRequest(pex_request: 'PexRequest', bin_names: 'Iterable[str]' = (), site_packages_copies: 'bool' = False) -> 'None'

WorkunitsCallbackFactory                     A wrapper around a callable that constructs WorkunitsCallbacks.  NB: This extra wrapping is because subtyping is not supported in the return position of a rule. See #11354 for discussion of that limitation.

ZipBinary                                    

_LoadedGlobalScalacPlugins                   _LoadedGlobalScalacPlugins(names: 'tuple[str, ...]', artifact_address_inputs: 'tuple[str, ...]')

_RelevantTargets                             _RelevantTargets(targets: 'FrozenOrderedSet[Target]')

_UserLockfileRequests                        

Use `./pants help $api_type` to get help for a specific API type.

$ ./pants help ThirdPartyPkgInfo

`ThirdPartyPkgInfo` API type
----------------------------

All the info and files needed to build a third-party package.

The digest only contains the files for the package, with all prefixes stripped.

Returned by 1 rule:

pants.backend.go.util_rules.third_party_pkg.extract_package_info
    activated by pants.backend.experimental.go

$ ./pants help TestSetup

`TestSetup` API type
--------------------

TestSetup(process: pants.engine.process.Process, results_file_name: Union[str, NoneType])

Returned by 4 rules:

pants.backend.python.goals.pytest_runner.setup_pytest_for_target
    activated by pants.backend.python

pants.backend.shell.shunit2_test_runner.setup_shunit2_for_target
    activated by pants.backend.shell
    Setup shunit2

pants.jvm.test.junit.setup_junit_for_target
    activated by pants.backend.experimental.java

pants.backend.scala.test.scalatest.setup_scalatest_for_target
    activated by pants.backend.experimental.scala

Granted, the help info is a bit terse, but that's only because there has not been a real motivation for providing it before. With this, any rule descriptions and doc strings, along with doc strings on output types is presented.

It's a nice place to discover what types are actually available.

~~Perhaps also required inputs should be presented for the rules?~~ Done.

kaos · 2022-01-05T15:53:52Z

Added input type info:

$ ./pants help Process

`Process` API type
------------------

Process(argv: 'Iterable[str]', *, description: 'str', level: 'LogLevel' = <LogLevel.INFO: 'info'>, input_digest: 'Digest' = Digest('e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 0), immutable_input_digests: 'Mapping[str, Digest] |
None' = None, use_nailgun: 'Iterable[str]' = (), working_directory: 'str | None' = None, env: 'Mapping[str, str] | None' = None, append_only_caches: 'Mapping[str, str] | None' = None, output_files: 'Iterable[str] | None' = None, output_directories:
'Iterable[str] | None' = None, timeout_seconds: 'int | float | None' = None, jdk_home: 'str | None' = None, execution_slot_variable: 'str | None' = None, cache_scope: 'ProcessCacheScope' = <ProcessCacheScope.SUCCESSFUL: 'successful'>, platform: 'Platform
| None' = None) -> 'None'

Returned by 6 rules:

pants.backend.python.util_rules.pex.setup_pex_process
    activated by pants.core
    2 inputs: PexProcess, PexEnvironment

pants.backend.python.util_rules.pex.setup_venv_pex_process
    activated by pants.core
    2 inputs: VenvPexProcess, PexEnvironment

pants.backend.python.util_rules.pex_cli.setup_pex_cli_process
    activated by pants.core
    6 inputs: PexCliProcess, PexPEX, PexEnvironment, PythonNativeCode, GlobalOptions, PexRuntimeEnvironment

pants.backend.shell.shell_command.prepare_shell_command_process
    activated by pants.backend.shell
    3 inputs: ShellCommandProcessRequest, ShellSetup, BashBinary

pants.backend.docker.subsystems.dockerfile_parser.setup_process_for_parse_dockerfile
    activated by pants.backend.experimental.docker
    2 inputs: DockerfileParseRequest, ParserSetup

pants.backend.go.util_rules.sdk.setup_go_sdk_process
    activated by pants.backend.experimental.go
    4 inputs: GoSdkProcess, GoSdkRunSetup, BashBinary, GolangSubsystem

kaos · 2022-01-05T16:15:52Z

Found this pretty example (and with advanced help, shows all awaitable get constraints too):

$ ./pants help-advanced BuiltPackage

`BuiltPackage` API type
-----------------------

BuiltPackage(digest: 'Digest', artifacts: 'tuple[BuiltPackageArtifact, ...]')

Returned by 6 rules:

pants.core.target_types.package_archive_target
    activated by pants.core
    1 input: ArchiveFieldSet
    6 gets: Get(Targets, UnparsedAddressInputs, ..), Get(FieldSetsPerTarget, FieldSetsPerTargetRequest, ..), Get(BuiltPackage,
        PackageFieldSet, ..), Get(HydratedSources, HydrateSourcesRequest, ..), Get(PySnapshot, PyMergeDigests, ..), Get(PyDigest, CreateArchive, ..)

pants.backend.python.goals.package_pex_binary.package_pex_binary
    activated by pants.backend.python
    4 inputs: PexBinaryFieldSet, PexBinaryDefaults, PythonSetup, UnionMembership
    3 gets: Get(ResolvedPexEntryPoint, ResolvePexEntryPointRequest, ..), Get(TransitiveTargets, TransitiveTargetsRequest, ..), Get(Pex,
        PexFromTargetsRequest, ..)

pants.backend.python.goals.setup_py.package_python_dist
    activated by pants.backend.python
    2 inputs: PythonDistributionFieldSet, PythonSetup
    6 gets: Get(TransitiveTargets, TransitiveTargetsRequest, ..), Get(DistBuildChroot, DistBuildChrootRequest, ..), Get(PyDigest,
        PyAddPrefix, ..), Get(BuildSystem, BuildSystemRequest, ..), Get(DistBuildResult, DistBuildRequest, ..), Get(PySnapshot, PyDigest, ..)

pants.backend.docker.goals.package_image.build_docker_image
    activated by pants.backend.experimental.docker
    5 inputs: DockerFieldSet, DockerOptions, GlobalOptions, DockerBinary, ProcessCleanupOption
    3 gets: Get(DockerBuildContext, DockerBuildContextRequest, ..), Get(WrappedTarget, Address, ..), Get(FallibleProcessResult, Process, ..)

pants.backend.go.goals.package_binary.package_go_binary
    activated by pants.backend.experimental.go
    1 input: GoBinaryFieldSet
    6 gets: Get(GoBinaryMainPackage, GoBinaryMainPackageRequest, ..), Get(BuiltGoPackage, BuildGoPackageTargetRequest, ..), Get(ImportConfig,
        ImportConfigRequest, ..), Get(PyDigest, PyMergeDigests, ..), Get(LinkedGoBinary, LinkGoBinaryRequest, ..), Get(PyDigest, PyAddPrefix, ..)
    Package Go binary

pants.backend.java.package.deploy_jar.package_deploy_jar
    activated by pants.backend.experimental.java
    3 inputs: BashBinary, ZipBinary, DeployJarFieldSet
    5 gets: Get(Classpath, Addresses, ..), Get(PyDigest, CreateDigest, ..), Get(ProcessResult, Process, ..), Get(PyDigest, PyMergeDigests,
        ..), Get(PyDigest, PyAddPrefix, ..)

    Constructs a deploy ("fat") JAR file by
    1. Resolving/compiling a Classpath for the `root_address` target,
    2. Producing a ZIP file containing _only_ the JAR manifest file for the `main_class`
    3. Creating a deploy jar with a broken ZIP index by concatenating all dependency JARs together,
       followed by the thin JAR we created
    4. Using the unix `zip` utility's repair function to fix the broken fat jar

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

kaos · 2022-01-05T17:39:56Z

Ready for review as #13993 has landed.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

[ci skip-rust]

benjyw

This is super, super epic. And we can generate docsite pages as well in a followup, I assume...

benjyw · 2022-01-16T19:19:16Z

src/python/pants/help/help_printer.py

+    def _print_all_api_types(self) -> None:
+        self._print_title("Plugin API Types")
+        api_type_descriptions: Dict[str, str] = {}
+        for api_type, rule_infos in self._all_help_info.rule_output_type_to_rule_infos.items():


In your example the output is sorted alphabetically, but I'm not sure how?

Perhaps we should omit types beginning with an underscore? Those are not intended to be used by general plugins.

Theres a "sorted" call in the returned dict comprehension of get_rule_infos.

Got it. So I suggest we filter out types that start with _, and then this is good to go.

kaos · 2022-01-16T22:21:36Z

Yes, that's the idea :)

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano

So cool! 😍

Eric-Arellano · 2022-01-18T20:25:56Z

src/python/pants/help/help_printer.py

+            print(f"{name}{description}\n")
+        api_help_cmd = f"{self._bin_name} help $api_type"
+        print(f"Use `{self.maybe_green(api_help_cmd)}` to get help for a specific API type.\n")
+
    def _print_global_help(self):
        def print_cmd(args: str, desc: str):


This should probably be updated to mention api-tools.

…refix (#14307) It's confusing that we use `Digest` everywhere in our docs, but the actual name is `PyDigest`. That becomes particularly confusing with `./pants help` now reporting on API types: #14081.

kaos added 5 commits January 5, 2022 18:38

Add api-types help info.

e70809e

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

reserve "api-types" name.

086d9ef

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

add input types to rule info.

4a0258a

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

cleanup inputs presentation.

21af509

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

show input gets when requesting advanced help.

3604452

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

kaos force-pushed the help_api_types branch from 2f32d4f to 3604452 Compare January 5, 2022 17:39

kaos requested a review from Eric-Arellano January 5, 2022 17:39

kaos requested a review from benjyw January 14, 2022 14:15

kaos added 5 commits January 14, 2022 15:16

Merge branch 'main' into help_api_types

c6473de

test rule help info extraction.

42e62e4

forward Subsystem help to the subsystem rule doc.

edcb56f

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Add help text to MockJvmTool.

aa9274d

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Merge remote-tracking branch 'upstream/main' into help_api_types

38c25de

[ci skip-rust]

benjyw reviewed Jan 16, 2022

View reviewed changes

kaos added 2 commits January 17, 2022 10:25

Merge branch 'main' into help_api_types

d0df7b4

review feedback.

bf1b37d

kaos requested a review from benjyw January 17, 2022 09:29

Fix test issue.

2964281

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

benjyw approved these changes Jan 17, 2022

View reviewed changes

kaos merged commit 33e341a into pantsbuild:main Jan 17, 2022

kaos deleted the help_api_types branch January 17, 2022 16:35

Eric-Arellano reviewed Jan 18, 2022

View reviewed changes

stuhood mentioned this pull request Jan 19, 2022

Generate API references for plugin #10945

Open

Eric-Arellano mentioned this pull request Jan 29, 2022

[internal] Rename filesystem types defined in Rust to not have Py prefix #14307

Merged

kaos mentioned this pull request May 28, 2022

Add API types and rules to cli help info #14227

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Help api types #14081

Help api types #14081

kaos commented Jan 5, 2022 •

edited

kaos commented Jan 5, 2022 •

edited

kaos commented Jan 5, 2022 •

edited

kaos commented Jan 5, 2022

benjyw left a comment

benjyw Jan 16, 2022

benjyw Jan 16, 2022

kaos Jan 16, 2022

benjyw Jan 16, 2022

kaos commented Jan 16, 2022

Eric-Arellano left a comment

Eric-Arellano Jan 18, 2022

Help api types #14081

Help api types #14081

Conversation

kaos commented Jan 5, 2022 • edited

kaos commented Jan 5, 2022 • edited

kaos commented Jan 5, 2022 • edited

kaos commented Jan 5, 2022

benjyw left a comment

Choose a reason for hiding this comment

benjyw Jan 16, 2022

Choose a reason for hiding this comment

benjyw Jan 16, 2022

Choose a reason for hiding this comment

kaos Jan 16, 2022

Choose a reason for hiding this comment

benjyw Jan 16, 2022

Choose a reason for hiding this comment

kaos commented Jan 16, 2022

Eric-Arellano left a comment

Choose a reason for hiding this comment

Eric-Arellano Jan 18, 2022

Choose a reason for hiding this comment

kaos commented Jan 5, 2022 •

edited

kaos commented Jan 5, 2022 •

edited

kaos commented Jan 5, 2022 •

edited