experimental_sibling_repository_layout seems to create remote execution actions with invalid working_directory/output relationships #13188

illicitonion · 2021-03-09T17:22:17Z

Description of the problem / feature request:

When testing out an unrelated feature, using remote execution (with buildbarn), I ran into issues where Bazel errored that it could not compile DumpPlatformClassPath.class because:

JavaToolchainCompileClasses tools/jdk/platformclasspath_classes/DumpPlatformClassPath.class failed: not all outputs were created or valid

I'm running in a workspace with name example_all.

As far as I can tell, this error is because on the Command we're setting working_directory: example_all but also prefixing the output_files with example_all/, for example the output_files for my execution is example_all/bazel-out/bazel_tools/darwin-fastbuild/bin/tools/jdk/platformclasspath_classes/DumpPlatformClassPath.class.

The remote execution API specifies that output_files should be relative to the working_directory, rather than relative to the input_root, so I believe this prefixing shouldn't be happening. See this section of the remote API spec:

https://github.com/bazelbuild/remote-apis/blob/0943dc4e70e1414735a85a3167557392c177ff45/build/bazel/remote/execution/v2/remote_execution.proto#L547-L551

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Run a buildbarn cluster, and try to build any Java with --experimental_sibling_repository_layout=true.

What operating system are you running Bazel on?

The Bazel client is on macOS, the buildbarn cluster is running Ubuntu Focal.

What's the output of `bazel info release`?

development version

If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

Checked out f182645 and ran bazel build //src:bazel

If I revert 7149f57 this error goes away (though others are introduced), so I'll cc @lberki who may have some thoughts :)

The text was updated successfully, but these errors were encountered:

lberki · 2021-03-10T08:46:24Z

This is probably because RBE has a bug where the output directory is interpreted relative to input_root instead of according to the spec, so this is what Bazel did.

@eytankidron, do I understand correctly that if Bazel makes it so that output file paths are relative to working_directory, it will keep working with RBE since it's been partially fixed there?

@coeuvre , can you take this?

eytankidron · 2021-03-10T14:07:50Z

@lberki that is correct. This bug is now fixed in RBE. Sending output paths relative to the working directory will now work.

ulfjack · 2021-03-10T17:26:46Z

Changing this now is a breaking change for remote execution for users that are using --experimental_sibling_repository_layout.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced. On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after cl/356735700. The reason test didn't fail is two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced. On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after cl/356735700. The reason test didn't fail is two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible. execRoot usages in RemoteCache are all replaced. On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after cl/356735700. The reason test didn't fail is two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows. Fixes bazelbuild#13188.

…y default. When --experimental_sibling_external_layout is set, use the parent directory of execRoot as input root and set working directory to the base name of execRoot. Paths of output files are relative to working directory. When --incompatible_remote_paths_relative_to_input_root is set, paths of output files are relative to input root. Introduce RemotePathResolver which is used to convert local paths to remote paths and vice versa. We should prefer this class instead of using execRoot directly whenever possible in remote module. execRoot usages in RemoteCache are all replaced. On Windows, shared action results cache for cl.exe across different workspaces causing header dependency checking to fail. This was initially fixed by bazelbuild#9172, but is broken after bazelbuild@24c980b. The reason test didn't fail before this change is that two builds from different workspaces do not share the cache since input files are relative to the parent of exec root which contains workspace name. This change fixes that by adding workspace name as platform property for action running on Windows. Fixes bazelbuild#13188. Closes bazelbuild#13339. PiperOrigin-RevId: 369168230

lberki assigned coeuvre Mar 10, 2021

lberki added team-Remote-Exec Issues and PRs for the Execution (Remote) team P2 We'll consider working on this in future. (Assignee optional) labels Mar 10, 2021

ulfjack mentioned this issue Mar 10, 2021

Are outputs relative to exec root or working directory? bazelbuild/remote-apis#127

Closed

coeuvre added P1 I'll work on this now. (Assignee required) type: bug and removed P2 We'll consider working on this in future. (Assignee optional) labels Mar 11, 2021

coeuvre mentioned this issue Apr 13, 2021

Remote: Use execRoot as input root and do NOT set working directory by default. #13339

Closed

bazel-io closed this as completed in b863dbf Apr 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

experimental_sibling_repository_layout seems to create remote execution actions with invalid working_directory/output relationships #13188

experimental_sibling_repository_layout seems to create remote execution actions with invalid working_directory/output relationships #13188

illicitonion commented Mar 9, 2021

lberki commented Mar 10, 2021

eytankidron commented Mar 10, 2021

ulfjack commented Mar 10, 2021

experimental_sibling_repository_layout seems to create remote execution actions with invalid working_directory/output relationships #13188

experimental_sibling_repository_layout seems to create remote execution actions with invalid working_directory/output relationships #13188

Comments

illicitonion commented Mar 9, 2021

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

What operating system are you running Bazel on?

What's the output of bazel info release?

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

lberki commented Mar 10, 2021

eytankidron commented Mar 10, 2021

ulfjack commented Mar 10, 2021

What's the output of `bazel info release`?

If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.