Wire up PathMapper in SpawnInputsExpander
#19718
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fmeum
force-pushed
the
6526-wire-up-spawninputsexpander
branch
from
8662630
to
333b562
Compare
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` in `SpawnInputsExpander`, which takes care of this for inputs to `Spawn`s executed in a sandbox or remotely. Constructs specific to Blaze, filesets and archived tree artifacts, are not covered by this change. Work towards bazelbuild#6526
fmeum
force-pushed
the
6526-wire-up-spawninputsexpander
branch
from
333b562
to
b7fdade
Compare
This was referenced Oct 4, 2023
github-actions
bot
added
awaiting-review
tjgq
reviewed
Oct 4, 2023
src/main/java/com/google/devtools/build/lib/analysis/actions/StrippingPathMapper.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/exec/SpawnInputExpander.java
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/exec/SpawnInputExpander.java
Show resolved
Hide resolved
tjgq
requested changes
Oct 5, 2023
src/test/java/com/google/devtools/build/lib/exec/SpawnInputExpanderTest.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/exec/SpawnInputExpander.java
Show resolved
Hide resolved
tjgq
approved these changes
Oct 5, 2023
tjgq
added
awaiting-PR-merge
github-actions
bot
removed
the
awaiting-PR-merge
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 9, 2023
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` with sandbox outputs logic for sandboxed and worker sandboxed execution. An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721. Work towards #6526 Closes #19719. PiperOrigin-RevId: 571950903 Change-Id: I4a9d15da578da11f8677279b09e0864b8c33e9fc
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 16, 2023
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` in `RemoteExecutionService`, which ensures that paths of inputs and outputs are correctly mapped before being sent off to the remote executor and mapped back to the correct local paths when downloading the results. An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721. Work towards #6526 Closes #19721. PiperOrigin-RevId: 573806130 Change-Id: Ibbd4ff641eb301d78f5ec54813e65788d786fcea
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PathMappers rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up thePathMapperused by a givenSpawninSpawnInputsExpander, which takes care of this for inputs toSpawns executed in a sandbox or remotely.Constructs specific to Blaze, filesets and archived tree artifacts, are not covered by this change.
An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721.
Work towards #6526