Plugins that are usable by both ImJoy and Napari

[sofroniewn]

Description

Based on some initial discussions between @royerloic and @oeway the idea was floated that Napari and ImJoy could try and share a plugin ecosystem. Both projects aim to make python based image analysis, including packages using deep learning and gpus, more accessible to the masses.

ImJoy is leveraging modern web technology and can be used in a browser whereas Napari is a python package that can be used as a standalone desktop app or with jupyter notebooks. Allowing for shared plugins that could easily be used by both tools could have massive benefits for the user and developer communities.

Here are some comments from @oeway the main ImJoy developer in the napari slack that could be used to get this discussion going:

@royerloic @sofroniewn I like the idea of having a common python plugin infrastucture, here is the current design of ImJoy plugin system:
Here is some ideas behind the plugin system of ImJoy:

1. every plugin is a single file with .imjoy.html extension (e.g. link), the annotation declarative scheme we used is a json config block in the plugin file (e.g. link). We use a relatively consistent format across plugins in different languages (Javascript, Python, potentially more, WIP: ImageJ scripts ).
2. complementary code/modules are provided through requirements (link). Tags (inspired by docker) can be used to specify different requirements for different branch of the same plugin (e.g. link), the user will need to choose which tag to be used during installation (e.g. link).
3. plugin files can be hosted anywhere which can be accessed through https url including Github/Gist/Gitlab, Dropbox or Google drive, and can be with https://imjoy.io/#/app?plugin=PLUGIN_URL.
4. The recommended way of deploying ImJoy plugins is through Github repo, any Github Repo can be transformed into a ImJoy plugin repo by adding a file called manifest.imjoy.json. ImJoy will render the list of plugins from a repo through https://imjoy.io/#/app?repo=GITHUB_USER/REPO_NAME (e.g. link and link). Another benifit of making a ImJoy repo is, you can then use shorter plugin URI format to refer to the plugin, e.g.: you can use: https://imjoy.io/#/app?plugin=oeway/Anet-Lite:Anet-Lite instead of https://imjoy.io/#/app?plugin=https://github.com/oeway/Anet-Lite/blob/master/anetLite.imjoy.html as described in “3”.
5. The same plugin URI scheme described in “3" and “4” can be used to describe the dependency of a plugin (e.g. link).
6. Each plugin can expose a set of api (at least setup and run) through api.export call (e.g. link), the same applies to different types of plugins. Any plugin can call exported api functions in another plugin, the call can be concurrent through async/await syntax in JS and Python
7. You can describe the inputs/outputs of a plugin with JSON-schema (https://json-schema.org/), it will be helpful for validating the inputs/outputs of a plugin and also for building a computational graph in the future. (e.g. link)
8. One thing we really like to push forward is the concept of functional plugins — computational plugins consist of only pure functions with no side-effect (https://www.sitepoint.com/functional-programming-pure-functions/). The advantage of functional plugins is enhanced reproducibility and it allows us to better parallelize the plugin execution in the future. Example of a functional plugin: link. Ideally we would like most of the computational plugins be functional. For the moment, we don’t have a way to validate whether the plugin is functional, it’s upto the developer to add functional to flags (e.g: link), we planed to show a functional badge for those plugin in the plugin list of ImJoy.

[jni]

Thanks for this and 👋 @oeway! FYI my colleague @VolkerH said your ImJoy demo was the highlight of NEUBIAS for him. =)

I am all on board for enabling ImJoy plugins in Napari. I agree with you @sofroniewn that it would be a massive benefit if we shared this infrastructure. So I am +💯 on Napari supporting the above format.

Having said this, I think that our needs might be simpler (since we have no intention at this time of supporting other languages?), and I also want to support a simpler spec that annotates functions with Napari types using stub files and then leaves the API crawling to us. Napari types includes anything that can correspond to a Layer: grayscale image, multichannel image, segmentation (label) image, coordinate list, coordinate and size list, polygon list, dataframe, etc. As far as I can tell, this works perfectly for the above-defined functional plugins, but is probably not general enough for more complex plugins (anything that adds functionality to the GUI, rather than process data). @oeway do you have any thoughts about using a simple typing stubs file for functional plugins, and would you object to supporting that in ImJoy? We could make these "image-types" instead of "napari-types". =)

[oeway]

Hi @jni , thanks for your support!

I think I will need to understand a bit more about how Napari and its plugin work. I think the stub files you mentioned should be equivalent to the JSON schema we used in ImJoy, it should be easy to convert them if you have a list of predefined Napari types. If necessary, of course we are happy to support it in ImJoy.

My concern more about whether the "image-types" can be easily serialized in order send through websocket. In ImJoy, we try to separate the UI part form the computational part by place them into different plugin, and then we can run the UI plugin on a thin client (e.g. mobile), then run the computational part on another powerful server, we used websocket to enable transparent RPC call between different plugins. In order to pass data from one plugin to another plugin, we will need to serialize them and send through websocket.

I will try to discuss this with @sofroniewn today.

[bkmartinjr]

It seems likely that a high-fps UI that uses plugins will require some sort of websocket bypass for the image data. I explored a trivial version of this (using a socket + mmap) here. Getting mmap into chrome is not possible, but getting it into electron was fairly simple.

[jni]

To clarify, image type definitions will be just for annotations, and we are only using standard types like NumPy arrays, lists, tuples, etc.

[oeway]

@bkmartinjr Right, for sure using websocket would not allow transferring large amount of data as you can do in Napari, and I would rather see this as a complementary aspect between ImJoy and Napari. The potential solution you mentioned are definitely interesting, and we do have a ImJoy desktop app built with Electron. However, we want to focus only on features which can be supported with a standard browser. And also, when we think some actual use cases, I think transferring large a mount of data from the backend to the browser UI maybe not necessary.

In ImJoy, we tend to think the data send to the web UI is a view of the data rendered from the Python backend. With this in mind, the information displayed on the screen is always limited to the screen resolution (say 4K resolution), and since we will typically process static data, high-fps is typically not needed. For example, if have 1TB volumetric images from light-sheet microscope, to display it on a web UI, we will use python to read from disk and render a 2D view or crop a small region, and send to the browser.

Even if someone wants to build real-time streaming application, we will be able to handle such cases with websocket, we have a demo plugin for displaying live video through Micro-Manager. For even more demanding applications, I am currently investigating the possibility of using webrtc which natively supported by modern browsers and also there is already a python library aiortc.

That said, in the current implementation of ImJoy, since all the plugins are sandboxed (e.g. each python plugin runs in its own process, or even its own conda virtual environment), all the api calls including data exchange between python plugins will go through the browser. I think it would be more meaningful to bypass some inter-plugin api calls if they are between python plugins running on the same host. Or even further, if the inter-plugin calls are between different host, we can use potentially use webRTC to enable peer-to-peer data exchange. This would be benificial when we want to use ImJoy interface to coordinate computational backends running on many different hosts.

As some suggestions to make the bridging between Napari and ImJoy easier, I think it is important for Napari to support headless mode to be compatible with remote servers, have a good separation between UI plugins and computational plugins (including functional plugins). In ImJoy, we essentially force the developers to make different plugins for UI and logic/computation code. From our experience so far, we noticed that there are actually many benefits, it works effectively as a regularizer for coding behavior since you cannot directly manipulate UI elements in the python plugin, and they have to think more about using a set of api to interact with the UI plugin. On the other hand, this increases the reusability of the UI plugin, for example, we have been using the same UI plugin with several similar plugins (e.g.: noise2self, anna-palm etc.)

If Napari can implement a similar RPC scheme as ImJoy does, then Napari plugin will be able to communicate and bridge with ImJoy and any plugins (including JS, python, and window plugins (example) in ImJoy, vise versa.

@jni for the image type, if you can limit to standard types, then it will be very straightforward to connect to ImJoy, we have supported all the primitive types and numpy array.

[jni]

I think it is important for Napari to support headless mode to be compatible with remote servers,

Part of the big picture for Napari is any processing should be recorded as valid Python code, and "Napari headless" is just Python.

If Napari can implement a similar RPC scheme as ImJoy does, then Napari plugin will be able to communicate and bridge with ImJoy and any plugins

I would call this a stretch goal, since we actually want to minimise sandboxing in the beginning, but it is a very good stretch goal. We can envision certain plugins requiring sandboxing, and for those we fall back on the same RPC scheme.

if you can limit to standard types, then it will be very straightforward to connect to ImJoy, we have supported all the primitive types and numpy array.

👍 awesome. The only exception to this is that medium-term we want to support dask arrays, see #103. Probably this would also be nice for you to support?

[oeway]

Part of the big picture for Napari is any processing should be recorded as valid Python code, and "Napari headless" is just Python.

This sounds awesome, this will make it very straightforward to run Napari script within a ImJoy plugin and use Napari as a backend. Since you support recording as ImageJ macro, it will be much easier to make ImJoy plugin based on Napari to support remote batch processing (e.g. on a cloud server).

I would call this a stretch goal, since we actually want to minimise sandboxing in the beginning, but it is a very good stretch goal. We can envision certain plugins requiring sandboxing, and for those we fall back on the same RPC scheme.

Good. On the other hand, in ImJoy every plugin is sandboxed into different processes, if Napari plugins is by default non-sandboxing, we can basically take it as complementary mode for ImJoy. Say if the developer wants fast/huge data exchange between plugins, then they should make plugins in Napari and do most of the task inside Napari, and then pass the processed results to other ImJoy plugins (e.g. display in the web interface). I really see this as a good complementary for both software.

One related question is, would you consider to support single plugin file format? I saw your hello world plugin, the equivalent single file format would be napari-hello-world.napari.html , you can still keep yaml file format, but it would be better for ImJoy to read if you can use JSON directly in the file. Rendering a plugin list (e.g. on a plugin repository website) would be also easier with JSON. A single file format has limited code capacity, but on the other hand encourage the developer to modularize and package their code, decouple from Napari/ImJoy and reuse it elsewhere. And also a single file makes it much easier to share the plugin.

👍 awesome. The only exception to this is that medium-term we want to support dask arrays, see Napari/napari-gui#103. Probably this would also be nice for you to support?

Sure, I would love to support dask arrays. I think a naive implementation would be expose all the dask array api functions through RPC, such that a window plugin runs in the browser can manipulate the array remotely, while keeping the actual logic operation in the plugin engine with whatever dask array libraries we use on the python side. This implementation will impose very minimal work on our side, but I am not sure if there is better way to support it. What will you suggest to implement to allow remote operations? @jni

[jni]

One related question is, would you consider to support single plugin file format? I saw your hello world plugin, the equivalent single file format would be napari-hello-world.napari.html , you can still keep yaml file format, but it would be better for ImJoy to read if you can use JSON directly in the file. Rendering a plugin list (e.g. on a plugin repository website) would be also easier with JSON. A single file format has limited code capacity, but on the other hand encourage the developer to modularize and package their code, decouple from Napari/ImJoy and reuse it elsewhere. And also a single file makes it much easier to share the plugin.

The short answer, is "probably". The intermediate answer is, "don't read too much into our existing code because Napari is at a much earlier stage than ImJoy, especially when it comes to the plugins."

The longer answer: my initial reaction is that I don't want html as the input format, and if we do single file plugins, the preferred format is a single, well-documented, typed Python file. All the functions not beginning with _ in that file are turned into the plugin.

However, as far as I can tell it should be straightforward to convert from that .py to your .html, with the module-level header string turned into the html description, and vice versa, so we could just write converters and have everyone be happy.

Incidentally, part of the reason I like the single .py file is that then you can directly import it, no processing required at all, and use "Napari plugins" from Python.

What will you suggest to implement to allow remote operations?

I don't have specific ideas on this yet. =)

[oeway]

Ok, good to know your thoughts on this.

To be clear, I am not suggesting to use single file format to replace your existing format in Napari, it's rather a way to make the two plugin systems interpolatable. There are pros and cons between different design choices, but if we can make both work without much efforts, as you said everyone will be happy.

Another thing I want to clarify is, even though we use the .html extension, the file format is not really the standard html format, we just use a few customized xml tags to separate different text blocks. We choose the .imjoy.html because we want to benefit from the default syntax highlighting of existing code editor (Atom/Github etc.).

On the other hand, it's indeed more standard way to make separate files, and make it importable directly. However, I am not sure how would you deploy and distribute your plugins, I think if you use separate files, your plugin will need an additional packaging step (for example zip it) or relying on github or pip in order to make it sharable through one file or one single url. I would argue that in many cases a single file is more straightforward than a zipped plugin package, a git repo or a pip package, it's more readable and more compatible with git versioning. Considering this, there could be actual benefit to support single file format. After all It's easy to support -- instead of using unpack a zip file, you use a plugin file parser to read blocks and save into separate files.

[GenevieveBuckley]

The architecture for napari plugins has matured a bit in the last couple of years, so I thought I'd update the comments here.

The npe2 (napari plugin engine 2) package governs how plugins are specified and interact with napari. Basically, plugins are an ordinary python library/package, and you add a napari.yaml config file which describes how it can interact with napari.

The manifest reference is here, and you can see a real world example of a napari.yaml file here.

To make it easier for users, there is a napari plugin cookiecutter provided, and Draga has made this excellent youtube tutorial.

[GenevieveBuckley]

Recapping some of the other points from upthread:

• napari uses yaml specification files, not json; and typically napari plugins will have multiple files.

• Also, napari isn't pushing functional plugins nearly as hard as ImJoy is. I can imagine a future where plugin authors might be encouraged to tag specific functions in the manifest as having functional behaviour. But for now, more or less anything goes as far as modifying data in place.

• 👍 awesome. The only exception to this is that medium-term we want to support dask arrays, see Support visualization of lazy ndarrays #103. Probably this would also be nice for you to support?
  Sure, I would love to support dask arrays.

  napari now supports dask arrays. So current supported array types include: numpy, dask, zarr, tensorstore (and I would like to see xarray in the future, but there's a bunch more work there that would need to happen first Consistent physical model of data and scene spaces #5949)