Diff against a dedicated exec version of the baseline configuration

[fmeum]

When --experimental_output_directory_naming_scheme is set to
diff_against_baseline, the current configuration is diffed against a
baseline and a hash of the diff is appended to the output directory.

If a target that is built in the exec configuration transitions further,
the diff would, before this change, include the (substantial) set of
flags that differ between the top-level target and exec configuration.
This made caching less efficient across builds that change flags that
are reset by the exec configuration, e.g. --collect_code_coverage.

This is fixed by comparing exec configuration against a dedicated exec
version of the top-level target configuration, which is safe since
is exec configuration itself is explicit in the output path even with
non-default values of --experimental_exec_configuration_distinguisher.

This includes a cleanup: Now that the host configuration is gone, fixed
changes to the CoreOptions when switching to the exec configuration are
better handled in CoreOptions#getExec.

Work towards bazelbuild/rules_go#3472

[fmeum]

@sdtwigg This is a small change that would help prevent certain cases in which diff_against_baseline could potentially be worse than the previous output directory suffix scheme across builds.

It is stacked on #16910, so please ignore the first commit.

[fmeum]

@com6056 Could you verify that this, together with bazelbuild/rules_go#3522, fixes the rebuilds for you?

[gregestren]

@sdtwigg This is a small change that would help prevent certain cases in which diff_against_baseline could potentially be worse than the previous output directory suffix scheme across builds.

By "previous" do you mean --experimental_output_directory_naming_scheme=legacy and --experimental_exec_configuration_distinguisher=legacy? (trying to remember which combination of flags we expect when!)

[com6056]

@com6056 Could you verify that this, together with bazelbuild/rules_go#3522, fixes the rebuilds for you?

Yep looks like both changes fix the issue, thanks so much, really appreciate it! 🎉

[fmeum]

@sdtwigg Rebased onto master, it's much smaller now.

[sdtwigg]

Heya! Sorry for the response delay here, thought I had responded earlier and was tagged in a way that I didn't get any more notifs to here.

The good news is that we actually discovered this issue internally and had been looking into ways to fix it. (There were some other considerations and time spent as to whether other native transitions need a similar treatment. The short story there is that the exec transition is likely 'special' here and the only one of major issue.)

The bad news is that I am a bit picky about the implementation here because I have my own internal version of doing this that I was still playing with and did not yet release. (Fun trivia: f7829f8 was done as part of writing those internal versions.)

In particular, we want to avoid adding new PrecomputedValue types so it would be preferable to instead get that post-exec baseline as a new SkyFunction call. Similarly, to accommodate future changes to the execution transition, would be best actually get a real ExecutionTransition via ExecutionTransitionFactory.createFactory and use that. Finally, to aid experimentation, was doing this by adding a new mode to --experimental_output_directory_naming_scheme called diff_against_dynamic_baseline.

To save you time, I am tempted to just submit my internal version instead unless you think one of the differences discussed above was irreconcilable. I do want to leave this open for a little bit longer to discuss what performance benefits users saw (as the main change to output directory calculation is still morally equivalent). In particular, I am especially curious if just these exec transition changes here cleaned up the performance issue without needing other cleanups to native transitions as this data would help me fast-forward some other tests I was doing.

[fmeum]

The approach you describe sounds more conceptual, which I like. I agree that you would be in a better position than me to apply that change, so please just go ahead - I will open a follow-up PR that switches Bazel over to the new mode when it becomes available.

For rules_go this change was everything to prevent rebuilds of our exec tools after ensuring that we do not leak coverage-related C++ flags into their builds (see bazelbuild/rules_go#3522).

[fmeum]

Closing in favor of #18561