Understanding overhead showing in Chrome trace

[kastiglione]

Description of the problem request:

We have a long final action at the end of our build that we'd like to speed up. When looking at a Chrome trace of our build, that final action has much more Bazel overhead than other actions in the build.

My reading of this is that the action (BundleTreeApp a python script in rules_apple) is ~6s, and then after that Bazel does some post-processing in ActionContinuation.execute and then yet more post-processing in the top level invocation (skyframe?).

In the above screenshot, ActionContinuation.execute appears to have 2s of post-processing "overhead", and "Bundling, processing and signing" has another 2s of post-processing "overhead".

Feature requests: what underlying problem are you trying to solve with this feature?

When looking at a Chrome trace: How can we understand how and why Bazel overhead exists, and what can be done about it.

We see overhead at the beginning of a build as well, but this issue is focused on how to understand performance factors for poorly performing actions like this.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

You'd need a sufficiently large iOS project that has a non-trivial .app bundle.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 2.0.0

[ghvg1313]

Don't know if this is related, but we are also experiencing similar overhead when calling build and test action separately (bazel build <target> then bazel test <target>) with tree-artifacts turned on.

Traces:
test trace with tree-artifacts turned off
test trace with tree-artifacts turned on

The overhead is not too bad considering this is only one target, but you can clearly see multiple action dependency checking distributed across threads before test runner executes, which, you don't see when tree artifacts turned off.
The performance of dependency checking gets significantly worse when there're multiple targets under test

[brentleyjones]

Looking into this, it seems the post processing is Bazel collecting the contents of the declare_directory?

[BalestraPatrick]

This is still a valid issue. Our iOS app bundle with 16K files takes over 75s to complete this action, even though the rules_apple BundleTreeApp action was optimized to only take 10s, we have no easy way to optimize the rest of time that Bazel spends in collecting the contents of all directories and files.

[chiragramani]

Yeah, we are seeing this issue as well.

• Linking took ~8.5 seconds but actuallyCompleteAction took 6.4 seconds. (this makes the overall linking action take ~15 seconds)
• Symlink creation takes significant time. (~12.5 seconds)
  

Any optimizations here would give us really good wins, especially for the incremental builds.

[meisterT]

@chiragramani do you have an example repository that we can try this with?

[brentleyjones]

git clone https://github.com/buildbuddy-io/rules_xcodeproj.git
cd rules_xcodeproj
git checkout bj/overhead-demo
cd examples/integration

bazel build //iOSApp

You can then look at the Linking iOSApp/Source/iOSApp_bin, Creating symlink bazel-out/applebin_ios-ios_sim_arm64-fastbuild-ST-4e6c2a19403f/bin/iOSApp/Source/iOSApp_lipobin, and Creating symlink bazel-out/applebin_ios-ios_sim_arm64-fastbuild-ST-4e6c2a19403f/bin/iOSApp/Source/iOSApp-intermediates/iOSApp_ExecutableName actions:

The blank space at the end of all of them scale with the size of the linked binary (so this example will be milliseconds, 73 in my case on each action, but on real projects it grows to multiple seconds, and is unnecessary overhead).

[wilwell]

Thanks @brentleyjones for reproduction code.
I was able to reproduce your example only on M1 machine.
I investigated Linking iOSApp... action. On that case the difference between ActionContinuation.execute and subprocess.run is sandbox.delete and sandbox.CopyOutputs, which is reasonable.

But actuallyCompleteAction following ActionContinuation.execute is more interesting. I'm not sure what exactly going there, but I suppose that this is some manipulations with output files, e.g. flush of ActionFileSysyem.

The profile is here.

[brentleyjones]

My concern is more the symlinks actions taking the same amount of extra time. Those shouldn't be sandbox copy and delete times, since it's just a symlinks. So I agree, the more concerning part isn't the sandbox but the actuallyCompleteAction portion.