What target modeling changes to land in Pants 2.8?

[Eric-Arellano]

The target generator proposal kicked off a slew of changes and new questions. This ticket tries to catalog Where We're At and Where We Should Go.

What has already landed

Much of the target generator proposal has been implemented:

• [internal] Refactor "file targets" / "subtargets" to use more generic Target Generation #12860
• [internal] Add new dir:generator#generated address syntax #12891, although the generated address syntax is only used by Go and Terraform, and it's still subject to change
• [internal] Make more explicit when adding dependencies on siblings for generated file-level targets #12906, which allowed for the fix Fix "file-level dependencies" on files and resources targets to not pull in sibling files #12941
• Go and Terraform both use target generation. Go has a "generating sources" field ([internal] Fix _go_internal_package complaining about missing .s files #13145)
• Fix ./pants dependencies --transitive when run on a target generator like python_library #12956
• Shell, Protobuf, and Java have split the "atom" target (java_source) vs. "generator" target (java_sources), although Shell and Protobuf are purely internal refactors for now.

What I suggest for Pants 2.8

Finish most of the original target generator proposal, but punt on any changes to project introspection.

• Finish internally splitting target generator vs. atom targets for files, resources, python_tests, and python_library. Plan to internally split python_library into two targets without user API changes #12954
• Rename targets we already have exposed:
  • python_requirement_library -> python_requirement
  • python_library -> python_sources
  • protobuf_library -> protobuf_sources
  • shell_library -> shell_sources
• Add a source: str field to complement sources: list[str].
• Rename the generated file targets to stop being the name of target generator
  • shell_source
  • shunit2_test
  • python_source
  • python_test
  • file
  • resource
  • protobuf_source
• Add an overrides field for file-based target generators like python_tests. This is a splashy new feature with tangible benefits in boilerplate reduction.
• Add ./pants mend to automate fixing this all.

Why do these now?

These changes can wait if I can't finish them in time—they're not blocking.

However, I'd like to try to land them because they clean up the conceptual space of "file targets" without changing any current semantics. The only deprecation is needing to run our migration script rename_target_types_pants28.py. We already have file targets, which are confusing and under-documented, both for users and for plugin authors. These changes give us the benefits of the target generator proposal right away.

I'd like to land the overrides field because it is an easy win to make a big dent in boilerplate and to give an alternative to the confusing semantics of how splitting out a new target currently requires excluding the sources from its previous owner.

We still have a lot of upcoming changes—landing these changes in Pants 2.8 will make those subsequent changes easier to adapt to. We're amortizing the deprecations.

What remains

• Replace our Python macros like python_requirements with target generators. Deprecation plan for macros (python_requirements and poetry_requirements) becoming targets with addresses #12915
  • Fixing this is high priority, but would formally commit us to the new generated address syntax, which we should wait until Pants 2.9+.
• Figure out what to do with the "file address" syntax. Deprecation plan for file address syntax #12917
• Fix modeling of cycle tolerance: Update cycle detection to not rely on "file targets" vs. "BUILD targets" #12871
• Decide if we want to support Address-less target generators, and what that looks like
• Deprecate directories being a shorthand for a target with the name left off Remove bad name= default for targets #12286
  • likely by first implementing the Address-less target generators proposal. Possibly, but not likely, by requiring name to be always set.
• Target generators should not depend on generated targets? #13118
• Fix project introspection like ./pants peek to include generated targets: [wip] Stop removing target generators from project introspection like ./pants list :: #12952 and ./pants list :: now includes generated targets #12951
• Figure out if we want to keep supporting target generators acting as aliases for generated targets: Proposal: get rid of UnexpandedTargets #13086

[stuhood]

Works for me. Thanks for writing this up.

• Fix project introspection like ./pants peek to include generated targets: [wip] Stop removing target generators from project introspection like ./pants list :: #12952 and ./pants list :: now includes generated targets #12951
• Figure out if we want to keep supporting target generators acting as aliases for generated targets: Proposal: get rid of UnexpandedTargets #13086

The fact that Specs don't match generated/file targets living in the matched directories is I think closer to being a bug than something we should improve. I expect that breaking that portion out of #13086 (which is overall more to do with what the dependencies of targets should be, rather than whether Specs match them) might be good.

[Eric-Arellano]

The fact that Specs don't match generated/file targets living in the matched directories is I think closer to being a bug than something we should improve.

Agreed that that is a super awkward edge, possibly enough to be a bug. Thanks for your comments on #13086 - I'm going to focus for now on what I proposed in Pants 2.8 because it is unblocked by everything else, and then will spend more time thinking about What Remains...

[Eric-Arellano]

I revised the plan for 2.8 to distinguish between renaming target types we already expose (python_requirement_library, protobuf_library) vs. giving a new target type name for generated targets (protobuf_source etc).

I think it makes sense to do the renames of existing symbols no matter what, as they are more descriptive (people get confused what "library" means) and it prepares us for introducing the new targets.

But I want to take more caution with the new target types, as it means users will be able to explicitly declare them in BUILD files. I want to make sure we have the modeling of those file-based targets right before we do this. For example, I feel fairly convinced that it should be file(source="f.json") rather than file(sources=["f.json"]) w/ eager validation that # files == 1. We can't make that type of change as easily once we expose the file target name.

[Eric-Arellano]

I had a productive weekend :) Status update:

Newly finished

• files and resources are now internally split between generator vs. atom
  • Rename ResourcesSources and split target into generator vs. atom #13194
  • Rename FilesSources and split target into target generator vs. atom target #13190
• Go has an overrides field, which sets up infrastructure for the file-based targets: [internal] Add overrides field to go_mod target generator #13187
• We now support a source: str field, vs. sources: list[str] Add source field for file-based targets #13202
• All target generators are renamed:
  • Deprecate python_library, protobuf_library, and shell_library in favor of python_sources, protobuf_sources, and shell_sources #13196
  • Deprecate python_requirement_library in favor of python_requirement #13133
• python_requirement target and its macros are updated to better reflect the generator vs. atom split:
  • Deprecate module_mapping and type_stubs_module_mapping fields for python_requirement in favor of modules and type_stub_modules #13197
  • Deprecate requirements_relpath and pyproject_toml_relpath in favor of source for python_requirements and poetry_requirements #13204

Remaining

• Split Python targets into generator vs. atom.
  • Will close Plan to internally split python_library into two targets without user API changes #12954
• Update file-based targets to use source: str.
• Rename file-based targets, e.g. to python_source.
• Add overrides field to target generators.
• Fix python_tests to generate python_source targets for .pyi type stubs and for conftest.py.
• Add ./pants mend to automate fixing all the changes.

[Eric-Arellano]

Almost everything I wanted to land in 2.8 is now landed!

Only last blocker is conftest.py modeling, only blocked by further review of #13274.

Newly finished

• Finished splitting target generators from atom targets, and renamed all targets to reflect this: python_sources and python_tests targets now generate python_source and python_test targets #13241
• Added a source: str field [internal] Use source field for protobuf_source, shell_source, shunit2_source, file, and resource #13226
• Added an overrides field: Add overrides field to python_sources and python_tests target #13270
• Decided we do indeed want to keep target generators acting as aliases for generated targets in most cases. Proposal: get rid of UnexpandedTargets #13086
• Fix dir: and dir:: to include generated targets #13263
• Pants's CLI output uses the full address dir:dir instead of the shorthand dir #13274
• Add ./pants update-build-files to automate fixing some deprecation warnings #13273

Remaining for Pants 2.9+

• Deprecation plan for macros (python_requirements and poetry_requirements) becoming targets with addresses #12915
• Figure out if we keep file address syntax, and possibly add directory address syntax Deprecation plan for file address syntax #12917
• Update cycle detection to not rely on "file targets" vs. "BUILD targets" #12871
• Decide if we want to support Address-less target generators, and what that looks like
• Deprecate directories being a shorthand for a target with the name left off Remove bad name= default for targets #12286
• Target generators should not depend on generated targets? #13118