Bazel rebuilds host-configured targets when $PATH changes even with --action_env=PATH=x

[mariusgrigoriu]

Description of the problem / feature request:

Bazel rebuilds host-configured targets when $PATH changes even when setting --action_env=PATH=x

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

EDIT: See #10291 (comment) for repro

What operating system are you running Bazel on?

$ uname -a
Darwin eris 19.0.0 Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64 x86_64 i386 iMac19,2 Darwin

What's the output of bazel info release?

release 1.2.0

[laszlocsomor]

I tried repro'ing this on Linux and macOS without success:

foo/BUILD:

genrule(
    name = 'x',
    outs = ['x.cc'],
    cmd = """echo -e '#include <stdio.h>\nint main() { printf(\"'$$PATH'\\\\n\"); return 0; }' > $@""",
)

cc_binary(
    name = 'y',
    srcs = [":x"],
)

I built using bazel build foo:y -s --action_env=PATH=/bin:/usr/bin, and watched for the SUBCOMMAND: entries in the output. Even if I changed the global PATH between builds, Bazel did not rebuild anything.

Are you building Go code? Can you show a repro? (I don't know what you have in //cmd/... and //pkg/...)

[mariusgrigoriu]

Looks like the pure go packages don't exhibit this issue and neither does the example you posted above. We have more targets in our build script than originally posted, so I will figure out which targets are involved...

All of the SUBCOMMAND: entries I'm seeing are all related to @jsonnet

So, does this mean each rule needs to implement --action_env=PATH=... correctly?

Edit: I was under the impression that --incompatible_strict_action_env from #6649 means that the PATH would no longer leak into the rules.

[laszlocsomor]

I don't know what this means, without understand what exactly breaks.

--incompatible_strict_action_env should do what you described.

[mariusgrigoriu]

Here's a repro that builds on top of your example.

BUILD:

load(":test_path.bzl", "test_path")

test_path(name="test_path")

genrule(
    name = 'x',
    outs = ['x.cc'],
    cmd = """echo -e '#include <stdio.h>\nint main() { printf(\"'$$PATH'\\\\n\"); return 0; }' > $@""",
)

cc_binary(
    name = 'y',
    srcs = [":x"],
)

test_path.bzl:

def _test_path_impl(ctx):
    return [DefaultInfo(files=depset([ctx.executable.exe]))]

test_path = rule(
    _test_path_impl,
    attrs = dict({"exe": attr.label(
        default = Label("//:y"),
        cfg = "host",
        executable = True,
        allow_single_file = True,
    )}),
)

If you build the cc_binary target directly as in your example, there's no rebuild even when changing the global PATH.

However, if you build the test_path target, you get a repro if you change the global PATH:

bazel build //:test_path -s --action_env=PATH=/bin:/usr/bin

Is this working as intended? Why would we see different behavior when building the cc_binary directly or if it is a dependency of a rule?

According to #6648 and #7026 --incompatible_strict_action_env defaults to true since Bazel 0.21.0. This contradicts the documentation and the output of bazel help build.

Adding --incompatible_strict_action_env to the above command, it no longer rebuilds:

bazel build //:test_path -s --action_env=PATH=/bin:/usr/bin --incompatible_strict_action_env

Was the --incompatible_strict_action_env flag reverted back to false?

[laszlocsomor]

Thanks for the repro!

I think the culprit is the "host" configuration. We've had similar bugs in the past, when the host config didn't respect properties that the target config did.

Yes, --incompatible_strict_action_env was reverted to false, because it broke too many things.

[laszlocsomor]

Looks like --action_env does not affect host-configured actions.

[philwo]

The --host_action_env= flag is available to modify the PATH for host-configured actions now.

[rosasco-wk]

Can you provide an example of how to set --host_action_env in .bazelrc that changes to PATH won't result in a full rebuild Phil ? I've tried both:
build --host_action_env=PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
run --host_action_env=PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

and similarly for action_env. I get a full rebuild every time PATH changes for host and cross-compiled targets.

[fmeum]

@rosasco-wk Do you have a reproducer? With the example in #10291 (comment) and either --action_env=PATH=something_fixed --host_action_env=PATH=something_fixed or --incompatible_strict_action_env, I can't reproduce this with Bazel 6.

[rosasco-wk]

I'm using bazel 5.2.0. I don't have this problem isolated. I'm observing it in the build of Android GPU Inspector which is a substantial build to set up given its dependencies.

Do you expect this behavior in 5.2.0 ?

[fmeum]

I don't think anything has changed in this area since 5.2.0, so I would expect that to work. If the project has many dependencies, is it possible that a repository rule picks up PATH and that causes widespread invalidation as a ripple effect?

[rosasco-wk]

Yes, that's possible but the degree of invalidation is hard to get my head around. PATH change is the same as bazel clean --expunge wrt number of targets affected. Is there a way using bazel query to determine which envs will affect a target and why ?