Extension Mechanism

[svenefftinge]

Theia should be extendable in the following ways.

• Both the frontend and the backend should be extendable
• To allow maximum flexibility we will allow extensions to run in the main processes and access everything. Defining APIs and their lifecycle is a separate topic.
• a user should be able to install and uninstall an extension at runtime

At the core of Theia we use dependency injection (inversify) to contribute functionality to existing hooks and also to provide new hooks. A plugin should contribute an inversify module, that is then just added on start up.

[svenefftinge]

If we want to allow all the flexibility of adding additional UI plugins in the form of Inversify modules, we will need to run webpack on the server side whenever a plugins gets added or removed and reload the application.

[hexa00]

We had a small brainstorm about the extension system, note that this is
just the result of a 1h informal talk so many ideas in there but more
though is required, anyway here's what we came up with:

We agree that:

• we need runtime extensions and core compile time extensions.
• the frontend and backend needs to be extensible.
• when an extension installs on a hook, the hook's owner must have a standard
  notification. IOW there must be a standard way for plugin discovery.
• extensibility is more important than the separation of the extensions
  from the core. However we need good tooling to identify problems.
  We need to investigate if there's a way to have extensibility but the ability to stop problematic extensions.

Open questions:

As far as we understand inversifyjs can be used to compose applications
at compile time and as such is a good way to extend the core to create
different IDEs.

However do not see how inversify could be used for runtime extensions ?
It seems like we need another system entirely.

Clarify differences between web frontend and electron. What is needed for
the web that is not needed for for electron and why ? (see last comment
from sven).

What would be a way to prevent a plugin to freeze the whole application
without serializing all the calls the the API ?

Backend should be able to install from a repo directly ? Or possibly
create a proxy to the frontend to download the files (in case the
backend doesn't have internet access).

Can extensions extends extensions ? with services ?

[svenefftinge]

Sorry, I missed your comment (should have read it before yesterday's meeting :)

I agree with the goals, you listed. But as mentioned yesterday, I think we can achieve this with compile time extensions.

I think we could have an extension-registry on the backend, that knows the current configuration (list of enabled extensions for frontend and backend) and is capable of rebuilding the application if the configuration changes. For the frontend it means we would run webpack with a new configuration. The client would need to reload the browser, afterwards.
For the backend we would only need to install and restart.

An extension could simply be an npm-package, with a certain tag (e.g. #theia-ui-extension and #theia-extension). The registry would then use npm-find and filter on these to show the user the available extensions. We download/install them through npm.

An extension package would specify a main entry file which exports one object (or an array, if we want to provide that) of a specific type (E.g. TheiaExtension), which has lifecycle call backs and provides inversify modules.

[Alspaladin]

I have a question about backend extensions, does the install and uninstall part mean that on the next server restart (or create after some strange state) the extensions should already be installed? So if the server will raise from docker image, the extensions should be already there? Or on every reboot we will install them again? If no, that means that we have to add parts of config to the dockerfile and for every client the image will be unique, not generic for everyone. Thinking of this, it's better to have one generic image with basic setup without extensions and blank dockerfile that extends it and is unique for every user. In this case we will be able to easily change the basic part without touching user config

…

On Apr 26, 2017 11:31, "Sven Efftinge" ***@***.***> wrote: Sorry, I missed your comment (should have read it before yesterday's meeting :) I agree with the goals, you listed. But as mentioned yesterday, I think we can achieve this with compile time extensions. I think we could have an extension-registry on the backend, that knows the current configuration (list of enabled extensions for frontend and backend) and is capable of rebuilding the application if the configuration changes. For the frontend it means we would run webpack with a new configuration. The client would need to reload the browser, afterwards. For the backend we would only need to install and restart. An extension could simply be an npm-package, with a certain tag (e.g. #theia-ui-extension and #theia-extension). The registry would then use npm-find and filter on these to show the user the available extensions. We download/install them through npm. An extension package would specify a main entry file which exports one object (or an array, if we want to provide that) of a specific type (E.g. TheiaExtension), which has lifecycle call backs and provides inversify modules. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/TypeFox/Theia/issues/28#issuecomment-297318144>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHBGkrm9N05H_gNpFvnGlxw30f2_hakvks5rzw8HgaJpZM4Mz2fa> .

[hexa00]

@svenefftinge Yes that sounds resonable for the installation of an extention at least. I'm not sure about enable/disable however seems like recompiling for that would be too slow. Could there be a way to allow that to be a runtime operation with the inversify way ?

@Alspaladin It seems to me that what someone wants to do in their docker config is their choice, having the extentions there or not.

The way I see it, the frontend has the master configuration for the extensions, not the backend so while you can save time by preinstalling common extentions in a docker container Dockerfile. Once the frontend connects it will install/uninstall what its configuration says should be there.

As an admin the best scenario I think is to have a blank Dockerfile, set the user home directory outside of the docker image and distribute a git managed recommended config to the users for the frontend.

That way kinds are kept generic for the admin and the users don't have to reinstall stuff all the time.

[hexa00]

Talking with Marc-Andre we had some concerns about API versions.

Say we release version 1.0 and extension Foo depends on a Theia interface defined in version 1.0
Fast forward to version 2.0 this interface has changed and the extension no longer compiles.

We would need to version the interfaces and have some kind of deprecation strategy to move the extensions to the new APIs and avoid non-compatible extensions to load.

Any ideas on how to do this ?

I just read in your 1st post " Defining APIs and their lifecycle is a separate topic." So I guess we could create that topic :)

[svenefftinge]

I would like to reuse npm as much as possible, so also for versioning. It is generally very weak compared to e.g. OSGi but it is the standard thing in JavaScript land.

For recompiling without further optimizations the current build time is 2 secs on my machine. Since we are talking about installing extensions, I think a couple of seconds to install them would be acceptable. That said there are plenty of possibilities to optimize. E.g. we could just use runtime module resolution, so the browser would send multiple requests for the individual extensions.

In general, a quick reload of an application seems to be much simpler than dealing with (un)loading of extensions and their services at runtime. The latter will require us to introduce and listen on all kinds of additional lifecycle events. I don't think that is worth it.

Re: docker. We should on no level make any assumptions on docker being used as the runtime.
When building a Theia application, you should be able to preinstall extensions but also, through an extension registry find additional extensions, which you can download. Those extensions will be downloaded to the backend where the extension registry runs. In the case you install a Theia app into a docker image it would be there and in desktop scenario it would be local. When you change the set of activated extensions, the client app will be recompiled and loaded into the browser window (refresh) if successful. Otherwise the user should be informed about any issues.

[hexa00]

@svenefftinge Right I had not though about npm so much but looking at node-semver it sounds quite flexible.

Re: recompiling, yes I agree I was just wondering if enabling / disabling could work a different way but I guess it will be the same as install/uninstall just without removing the actual extension files.

[hexa00]

The way I see it, the frontend has the master configuration for the extensions, not the backend

Just to update talking with others, I don't think so anymore, the backend should have the master config, given that it doesn't make sense for the web app front end to have such a config...

[hexa00]

Just to document a discussion we had with @svenefftinge we could have a first step where the extensions are specified in a config file and the app is manually recompiled.

This would make the development of extensions possible with the packaging we ultimately want with less of an effort than the full registry solution.

[akosyakov]

Some npm packages which can be useful:

• nodemon is often used for the auto server restarting
• pm2 seems to be very popular for all auto restarting, clustering, logging, monitoring and so on

[akosyakov]

I am going to use npms to look up packages instead of npm search and http://npmsearch.com/. npm search fails to provide up to date information and npmsearch.com returns everything that matches at least one of search criteria.

[akosyakov]

Confirmed by npmjs support that there is a delay about 24h in search API + advises how to index ourself: https://github.com/npm/registry/issues/198#issuecomment-319213770

I will still go with npms since it looks like they already do indexing but in the long run we can look into suggested indexing alternatives.

[svenefftinge]

Fixed with #391