Allow actions to opt into path mapping via execution requirements #19717

fmeum · 2023-10-04T09:16:14Z

This provides a simple and backwards compatible way for actions to experiment with path mapping:

rules authors can add supports-path-mapping to the execution_requirements of a Starlark action
end users can use tags and --modify_execution_info to opt individual targets or actions in and out of path mapping.

Also includes StrippingPathMapperTest, which wasn't included in the merge of #18098, with an additional test for a Starlark rule that opts in via the mechanisms described above.

Closes #19526
Work towards #6526

aranguyen · 2023-10-04T14:22:10Z

@fmeum I just realized the test wasn't included in the last merge. The patching tool failed and I had to manually resolve the differences and missed the test. Sorry about that.

I saw a few different cls, could you please share here their priorities and the order I should be reviewing them in? THanks!

src/main/java/com/google/devtools/build/lib/analysis/actions/PathMappers.java

aranguyen · 2023-10-04T14:50:22Z

src/main/java/com/google/devtools/build/lib/analysis/actions/PathMappers.java

+      OutputPathsMode outputPathsMode, String mnemonic, Map<String, String> executionInfo) {
+    if (outputPathsMode == OutputPathsMode.STRIP
+        && (SUPPORTED_MNEMONICS.contains(mnemonic)
+            || executionInfo.containsKey(ExecutionRequirements.SUPPORTS_PATH_MAPPING))) {


@fmeum Just thinking out loud here, I wonder if we should only allow the opt-in behavior for Bazel only because within google, we probably don't want users to flip path mapping on and off as it would cause forking (mapped and unmapped paths) and potentially harm caching? @gregestren WDYT?

Sure, I could see that make sense. I will leave that up to you, happy to add this here.

If doing so isn't too much work, can we please add the check then?

I thought I could use isThisBazel, but that is only available in tests. I don't see an easy and clean way to tell apart Bazel from Blaze. I could potentially look at the execpath of the output artifact of the action, but that seems hacky. Are you aware of a better way to implement the check?

I thought there was something similar to JavaClassPathMode.BAZEL but i don't think there is. I can check with others tomorrow. This shouldn't block this pr though. I will work on the import and work on the next one. Thanks!

fmeum · 2023-10-04T18:04:53Z

@aranguyen For Greg and you, only the current PR and then #19723 should matter. #19718, #19719, and #19721 can be reviewed by other folks, I just CCed you on them. After those, it's just end-to-end tests (#18155).

This provides a simple and backwards compatible way for actions to experiment with path mapping: * rules authors can add `supports-path-mapping` to the `execution_requirements` of a Starlark action * end users can use tags and `--modify_execution_info` to opt individual targets or actions in and out of path mapping. Also includes StrippingPathMapperTest, which wasn't included in the merge of bazelbuild#18098, with an additional test for a Starlark rule that opts in via the mechanisms described above.

aranguyen · 2023-10-05T01:40:43Z

@aranguyen For Greg and you, only the current PR and then #19723 should matter. #19718, #19719, and #19721 can be reviewed by other folks, I just CCed you on them. After those, it's just end-to-end tests (#18155).

Thanks for clarifying. We do need all of these pr merge for path mappings to work in bazel right?

fmeum · 2023-10-05T07:31:05Z

@aranguyen For Greg and you, only the current PR and then #19723 should matter. #19718, #19719, and #19721 can be reviewed by other folks, I just CCed you on them. After those, it's just end-to-end tests (#18155).

Thanks for clarifying. We do need all of these pr merge for path mappings to work in bazel right?

Yes, that's correct. #19723 isn't required for just Java support though, only for good Starlark support.

tjgq · 2023-10-05T12:07:59Z

Drive-by comment; apologies if this makes no sense. Now that we're adding an execution requirement, should we consider explicitly tagging the Java actions with it instead of making a decision based on the action mnemonic? Mnemonics are not unique and this opens us up to future cleanup work (i.e., someone reuses a mnemonic, gets path mapping behavior, and then we can't take it back because it's a breaking change).

fmeum · 2023-10-05T13:06:13Z

Drive-by comment; apologies if this makes no sense. Now that we're adding an execution requirement, should we consider explicitly tagging the Java actions with it instead of making a decision based on the action mnemonic? Mnemonics are not unique and this opens us up to future cleanup work (i.e., someone reuses a mnemonic, gets path mapping behavior, and then we can't take it back because it's a breaking change).

This makes a lot of sense and is a very logical cleanup step, it's just something that a Googler will have to do as I can't test the change and probably don't have visibility into all affected rules.

src/test/java/com/google/devtools/build/lib/analysis/actions/StrippingPathMapperTest.java

aranguyen

updated commit looks good. Internal tests are passing

fmeum requested review from a team and lberki as code owners October 4, 2023 09:16

fmeum requested review from katre and removed request for a team October 4, 2023 09:16

github-actions bot added awaiting-review PR is awaiting review from an assigned reviewer team-Configurability Issues for Configurability team team-Performance Issues for Performance teams team-Rules-Java Issues for Java rules labels Oct 4, 2023

fmeum requested review from gregestren, a team and aranguyen and removed request for lberki, katre and a team October 4, 2023 09:16

fmeum force-pushed the enable-starlark-path-mapping branch from cb9dc94 to 3407cec Compare October 4, 2023 09:19

fmeum mentioned this pull request Oct 4, 2023

Automatically map paths in Args#map_each #19723

Closed

aranguyen reviewed Oct 4, 2023

View reviewed changes

fmeum force-pushed the enable-starlark-path-mapping branch from 3407cec to fb56b64 Compare October 4, 2023 18:07

aranguyen reviewed Oct 5, 2023

View reviewed changes

src/test/java/com/google/devtools/build/lib/analysis/actions/StrippingPathMapperTest.java Show resolved Hide resolved

aranguyen reviewed Oct 5, 2023

View reviewed changes

src/test/java/com/google/devtools/build/lib/analysis/actions/StrippingPathMapperTest.java Outdated Show resolved Hide resolved

Adapt tests for Blaze

40a94e2

fmeum requested a review from aranguyen October 5, 2023 15:03

fmeum force-pushed the enable-starlark-path-mapping branch from fb56b64 to 40a94e2 Compare October 5, 2023 15:04

aranguyen approved these changes Oct 6, 2023

View reviewed changes

copybara-service bot closed this in a239ea8 Oct 9, 2023

github-actions bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Oct 9, 2023

fmeum deleted the enable-starlark-path-mapping branch October 9, 2023 14:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow actions to opt into path mapping via execution requirements #19717

Allow actions to opt into path mapping via execution requirements #19717

fmeum commented Oct 4, 2023 •

edited

Loading

aranguyen commented Oct 4, 2023

aranguyen Oct 4, 2023

fmeum Oct 4, 2023 •

edited

Loading

aranguyen Oct 4, 2023

fmeum Oct 4, 2023

aranguyen Oct 5, 2023

fmeum commented Oct 4, 2023

aranguyen commented Oct 5, 2023

fmeum commented Oct 5, 2023

tjgq commented Oct 5, 2023

fmeum commented Oct 5, 2023

aranguyen left a comment •

edited

Loading

Allow actions to opt into path mapping via execution requirements #19717

Allow actions to opt into path mapping via execution requirements #19717

Conversation

fmeum commented Oct 4, 2023 • edited Loading

aranguyen commented Oct 4, 2023

aranguyen Oct 4, 2023

Choose a reason for hiding this comment

fmeum Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

aranguyen Oct 4, 2023

Choose a reason for hiding this comment

fmeum Oct 4, 2023

Choose a reason for hiding this comment

aranguyen Oct 5, 2023

Choose a reason for hiding this comment

fmeum commented Oct 4, 2023

aranguyen commented Oct 5, 2023

fmeum commented Oct 5, 2023

tjgq commented Oct 5, 2023

fmeum commented Oct 5, 2023

aranguyen left a comment • edited Loading

Choose a reason for hiding this comment

fmeum commented Oct 4, 2023 •

edited

Loading

fmeum Oct 4, 2023 •

edited

Loading

aranguyen left a comment •

edited

Loading