Investigate copy_to_directory performance

[kormide]

See https://bazelbuild.slack.com/archives/CA31HN1T3/p1665178148461539.

A copy_to_directory on 2500 files is taking on the order of 10s. Investigate what's causing the slowness and see if there are any optimizations that can be made.

[jwnx]

I'm experiencing the same problem. Copying 513 files takes consistently ~2s.

[matthewjh]

I am also having problems, but with analysis-phase performance of this rule. Our project is a graph of various ts_project, npm_package_link, npm_package, and js_library targets. When making a change to a BUILD file, such as to add a dep, the Bazel process uses 500% CPU for 15-20s, in the analysis phase (so before execution).

Profiling shows that most of this time is spent in copy_to_directory_action:

[gregmagolan]

Should be improved with 350408b that is included in the 1.16.0 release

[matthewjh]

Thanks, @gregmagolan , analysis performance is good now. 🙌

Execution performance is still problematic, though, which is what the OP was about. Are you guys still planning to investigate this? Should we open a new issue or re-open this one?

[matthewjh]

It seems to me that, besides simplifying the logic, one thing that would certainly improve perf for larger directories is performing the logic in parallel on different subtrees. At the moment this rule males very poor use of CPU as everything is serial and blocking. This should be easy to implement (eg in js or go) but I'm not sure how to do it in bash alone.