Are outputs relative to exec root or working directory?

[tomlu]

The documentation for output paths makes several references to output paths being relative to the action's "working directory". I interpret this as outputs being relative to the "working directory" as specified in the command.

Example:

exec root: "/foo"
working dir: "bar"
output: "baz.out"

absolute path of output "/foo/bar/baz.out"

However, after reading the Go client in remote-api-sdks, I am pretty sure that this code treats outputs as being relative to the exec root, not the working directory of the command.

So either they are relative to the command's working dir, in which case there's a of bug to be filed against remote-api-sdks, or they are relative to the exec root, in which case the proto documentation should be fixed to remove the ambiguity.

[ola-rozenfeld]

All paths are relative to the exec root. You're right, the proto documentation should be fixed! Thank you!

[juergbi]

All paths are relative to the exec root. You're right, the proto documentation should be fixed! Thank you!

This contradicts the previous clarification in #26

BuildStream, BuildGrid and BuildBarn (buildbarn/bb-remote-execution#18) follow the current proto documentation. I.e., they treat output paths as being relative to working_directory and this change would break existing clients and servers.

[ola-rozenfeld]

Bleh! Sorry, Juerg! You're right. I have short memory :-)
Okay, then this should be fixed on the RBE side, and then the SDK...

[tomlu]

So the conclusion is that the documentation is correct, our RBE implementation is wrong, and the SDK is (consequently) also wrong?

What would the migration path be for clients that rely on the previous behaviour (relative to exec root)?

[ola-rozenfeld]

I think that RBE will have to go through a phase where it will infer the client assumption based on where the actual outputs of the action are on the worker, before all its clients are fixed.
@bergsieker @EricBurnett

[tomlu]

That sounds touchy. Could we be explicit with some sort of RBE-specific migration bit in the protos, clearly marked as RBE only?

[ola-rozenfeld]

I don't think it's that touchy. To my knowledge, RBE only has two clients using the feature. Importantly, Bazel is not one of them! So even though the simple migration plan is not formally well defined (what if an action creates two outputs with the same path, one relative to exec root and one relative to work-dir?) I suspect it will work well in practice. And we can quickly fix the two clients, so that the phase doesn't need to last more than a couple of weeks.
And if we do see cases of ambiguity, we can flag those and reevaluate...
@bergsieker and @EricBurnett -- thoughts?

[tomlu]

I am working on such a client right now, and maybe the Chrome guys are as well. (Unless that's one of the ones you were aware of?)

How can you know whether an action creates outputs relative to the working dir and exec root? After the action completes, do you look for both of them, then adjust the output accordingly?

[ola-rozenfeld]

Yes, that is what I'm suggesting. As a temporary fix in RBE, look for both paths, take the one that exists, and log/surface any inconsistencies (such as both exist, or one output exists under working dir only while another under exec root only).

[EricBurnett]

@ola-rozenfeld your proposal SGTM. Or depending on how launched these clients are (is the use of our mis-implementation in prod, or just in progress?), we could possibly just make a backwards-incompatible (breaking) change to RBE and accept a short period where these clients don't work, in the interest of getting to the right behaviour sooner.

In either case, I suggest we take this discussion to a RBE bug - sounds like the API is clearly well defined and we've simply mis-implemented it, which is a RBE issue and not a remote-apis issue :).

[sstriker]

@EricBurnett , assuming discussion moved to an RBE bug, and no changes are needed here, can this be closed?

[bergsieker]

Yes, we've moved this to an internal bug. The published spec is clear, and we'll update our server to match.

[ulfjack]

Also see bazelbuild/bazel#13188; Bazel wasn't changed to match, although it's only assuming the old semantics if an experimental flag is set.

[ulfjack]

The code in Bazel that relied on this was added on Feb 13, 2020.