Improve performance of building Elasticsearch modules #76726
Description
The ML native code dependency we use contains native binaries for all supported platforms (Windows, Linux, Mac on both x86 and aarch64). When we actually create the ML module bundle, we indiscriminately include all of these, and then later filter out inapplicable ones when actually building the appropriate distribution. The problem is when generating a local distribution for testing, we end up packaging all these files for no reason. On my local machine, just building the ML bundle alone is a 16s operation. It accounts for over half of the time of building a local distro. We can do better here. A few options:
- Produce multiple variants of the plugin and we grab the right one depending on which distro we're building. This is likely to be a big pain and introduce unneeded complexity though.
- Don't "bundle" modules anymore. We should do something similar to what we did when we refactored distribution building to no longer create an archive, and instead just provide a directory as an artifact. This way we avoid zipping, then immediately unzipping a ton of artifacts for no reason. We'd still have to unpack the ML binaries, but maybe we could implement that as a cacheable artifact transform?