Remove the optimizer from builds

[epixa]

The optimizer runs in kibana releases whenever you install a plugin and whenever you start kibana after removing a plugin. This shouldn't be necessary or even possible.

The bundles that get created at build time and shipped with the core release should be immutable, and all plugins should need to create their own bundles during their own build process. Kibana should not in any way attempt to wrangle the actual plugin code itself, and plugins should not in any way be concerned with Kibana doing so.

This should mean that the optimizer does not get shipped with builds and is only used in development. That will greatly speed up plugin install/removal, will improve reliability of plugin installs and the integrity of the core kibana build, remove the need for certain permission wrangling that occurs now for package installs, and will allow us to ship Kibana with defaults for low max memory usage.

This should also free us to explore changes to the entire dev build system to make it faster and easier to use since we won't be burdened by production requirements of the optimizer. Ultimately, it will allow us to treat our tooling around the dev process as simply a part of the project rather than as part of the product itself.

Blockers:

• Status page UI (Migrate Status Page UI #67979)
• ui/styles (Migrate ui/styles to New Platform #58369)
• UiSettings injected via legacy metadata (Migrate injected UiSettings from legacyMetadata #22779)
• [build] kbn/optimizer needs ran from source [build] kbn/optimizer needs ran from source #71881

Tasks:

• Disable the legacy optimizer in development
• Disable legacy optimizer in production
• Remove clustering mechanism
• Remove src/optimizer and related dependencies
• Remove karma & related CI tasks(Tim Roes)

Closes #19633
Closes #19678
Related to #57406

[epixa]

/cc @tylersmalley

[spinus]

Would love to have that.

[epixa]

I removed the breaking label because this isn't necessarily a breaking change.

[epixa]

Once this is done, we can just close #5071 if that's still open.

[bradvido]

Will this also fix #19678 ?

[jinmu03]

https://github.com/mistic/webpack-dll-spike-app
https://github.com/mistic/webpack-dll-spike-plugin
https://github.com/mistic/webpack-dll-spike-app/tree/master/src/code/bundler

[mistic]

I have some updates on this matter. Over the past days I spent some time investigating ways to kill/drastically reduce the use of optimizer on Kibana. I have also spent some time in a POC you can found here mistic/webpack-dll-spike-app.

Removing the optimizer at this stage, if possible, will be very difficult as we in the current architecture we hardly rely on it. However I (and other folks on the @elastic/kibana-operations ) believe that could be possible to follow another path, that instead of remove the optimizer will help us to drastically reduce the use of it and move us in a step further into the goal of removing it! The main ideas behind this possible solution rely on the webpack DLL (I will narrow the description to a production production situation):

1st Step - Kibana Side - Building Kibana

• Add a DLL bundler to the optimizer with 2 main pieces: a compiler and a loader.
• In the first stage (during our CI pipeline) the compiler runs and builds the needed DLLs for Kibana (vendors_client, vendors_server, kibana_client, kibana_server).
• The loader kicks in and generate the main entry bundles referencing the generated Dlls and generating a first main entry bundle (this should be super fast!).
• Kibana gets released with the generated assets, the dlls and the main entry bundle, as long as with the bundler itself.
• When started in prod Kibana will always referencing all the Dlls it haves and regenerating the main entry bundle, then start.

2nd Step - Plugin Side - Building a Plugin

• For the plugins the flow is the same. The plugin template ships the DLLBundler and downloads the Kibana DLLs for a specific version.
• Runs the compiler to generate the plugin DLLs, and referencing the Kibana DLLs.
• The plugin gets distributed as form as a DLL.

3rd Step - Kibana Side - Importing a Plugin

• In order to finally importing a plugin, it is just need to run a plugin-installer to download the plugin DLL and put it in the kibana Dlls folder.
• Re-run the Bundler Loader to regenerate the main entry files
• Restart kibana server and we are done! (another approach could be follow like for example spin up a second kibana server and kills the older when the new is started)

We believe that this could be a possible solution for the problem, and it will also allow us moving in the direction of being able to remove the optimizer from production. You can found some of the concepts in the above mentioned POC. As the first step we decide trying to implement a vendors dll, only for the client side, and then re-evaluate! A new issue #20749 was created to accomplish it.

[joshdover]

Remaining plugins that still rely on the optimizer in production:

• Timelion plugin (Migrate timelion to the NP. #69160)
• Status page plugin (Migrate Status Page UI #67979)

We also have the "tests_bundle" plugin which is used for building a bundle of tests to be executed inside a real browser using Karma. I believe we should be able to remove the optimizer from production builds before we have a replacement mechanism for this. However, it may be simpler to migrate all of this first and just completely remove the legacy optimizer rather than attempting to have special prod-only behavior.

Virtually all the remaining Karma tests are in the kibana plugin. We have some in ui/public but these should be safe to delete completely. The Kibana App team has an issue for converting these tests over to jest (#57813). @timroes if we get to a point soon where this is the last blocker to removing the legacy optimizer, how comfortable would we be with disabling these tests until your team has time to migrate them to jest? Do we see breakages in these visualizations often?

[timroes]

Just to leave this in written state here: We're working on migrating the Karma tests really soon all over to other test frameworks. If there are still remaining tests left after 7.10, we're fine disabling/removing them until we can migrate them to a different test framework.