Plugin deployment slows down Theia frontend startup time

[sdirix]

Summary
When using VS Code extensions the Browser frontend startup time increases when started shortly after the backend. This is a typical use case for online cloud deployments. The same scenario tested in Theia-Electron did not result in a noticeable increase of the startup time. Some further investigation is needed to understand whether this issue has an impact on Electron deployments.

Description
For this issue the frontend is considered started up once the loading indicator is replaced with the actual Theia UI. The times were measured locally using the performance script introduced in #9777.

In our tests the frontend startup time increased on average by 37% when adding additional VS Code extensions to the built-in plugins. In concrete numbers the startup time increases from about 2,7 seconds only using the builtin plugins to about 3,7 seconds with the built-in plugins plus the the following extensions: clangd, Kubernetes, CMake Tools, CMake, remove empty lines, Gitblame and Native Debug

The later the frontend is started, the lesser the startup time increases. There is no noticeable increase in startup time when started at least 5 seconds later than the backend.

When profiling the frontend, we noticed the following: When looking at the time passed between the dom-loaded and the onLoad event, it can be seen that the requests to the backend take much longer.

Frontend startup profiled while backend is busy:

Frontend profiled during regular backend usage:

It's noticeable that when the backend is bust, all requests take a lot more time until they are resolved, summing up to a total of 1 second delay on average.

We debugged the backend and determined that this is related to the “deploy” step of the extension mechanism. When this deploy step is manually delayed (we just used a timeout of multiple seconds) the frontend experiences no delay in its requests.

Potential solutions

We can think of multiple approaches to solve or partially negate this problem:

1. Investigate the backend process and determine (and potentially fix) the cause of the delay when requests are handled
2. Reducing the amount of requests from the frontend to the backend to lessen the impact of delayed answers.
3. Pausing the “deploy” mechanism on the backend when a frontend is in its startup phase
4. Offer a mechanism by which Theia applications can delay extension deployments so that a quickly started frontend can startup faster.

Option 1 is the most preferable one. Option 2 can significantly reduce the delay, however it probably needs some invasive changes to the build system and frontend architecture. Option 3 needs to be investigated in terms of effort. Also together with Option 4 it shares the downside that it’s also good for the user experience when extensions are started up as fast as possible, so that they are sooner available.

[tsmaeder]

Very interesting investigation, I wouldn't have expected the "deploy" phase to take that much CPU. Can you affirm that no plugins are activated during the startup phase?

[sdirix]

Very interesting investigation, I wouldn't have expected the "deploy" phase to take that much CPU. Can you affirm that no plugins are activated during the startup phase?

Do you mean Theia plugins and VS Code extensions? They are synced, loaded and started only after the "deploy" step is completed. In our measurements the frontend was already loaded before the deployment step finished, so the plugins themselves should not be able to interfere here.

To give an overall impression: When using nine VS Code extensions, we measured ~8 seconds for the deployment step, while syncing, loading and starting took ~1.5 seconds combined.

[tsmaeder]

Does the time for deployment change the second time we're starting the back end? The plugins should only be unzipped once to a stable location.

[sgraband]

I should note that the numbers above were measured with .vsix files in the plugin folder. It seems like in that case, the file needs to be unzipped every time the backend is started.

Maybe this should be changed?

We also tested it with unzipping the .vsix files in the plugin folder. Then the startup time goes down to 3.1 seconds, and the Deploy time is only around 6 seconds. The phenomenon described by the screenshots still occurs, just on less requests as the deployment step is a bit shorter.

When defining the plugins using the theiaPlugins tag in the package.json the behavior is the same as with unzipped files in the plugins folder.

So unzipping the files in the plugins folder lessens the impact of this issue, but it should still be investigated why the requests are taking longer than expected, when the backend is busy.

[tsmaeder]

the file needs to be unzipped every time the backend is started.

that is odd: there is code in there somewhere that should skip that step if the unzipped contents are already present.

But still, 3.1 seconds seems like a lot of time.

[sgraband]

If you put .vsix files into the plugins folder they are actually never unzipped in the plugins folder. This seems to happen in a temporary directory.
Maybe this could be improved so that the plugins folder is checked for zipped extension and unzips them to avoid this regression in startup performance on future runs.

[sdirix]

There is one important detail missing from the original observation: The influence of the backend on the startup time must also be measured in production mode. There the frontend is (mostly) bundled to a single module instead of being split into dozens of separate scripts. The impact of a busy backend might therefore be much less in production mode as there is a single initial request to serve, instead of many consecutive requests.