Using neuroglancer as an embedded headless renderer

[Le0C]

Hi all - I thought I would start a discussion on this topic, as there is an open PR #1072 which touches on some features that are related.

I'd like to explore whether the CommandCatalog introduced in #1072 could become the single, neuroglancer-native place where the viewer's available commands are both enumerated and registered — with the command palette, the keyboard bindings, and any host-built control surface all being consumers of it, rather than each maintaining its own list.

Firstly some context, I am using neuroglancer as a 3d tomographic viewer for rendering muon tomography scans at GScan. Whilst our domain is quite far from the neuroscience/connectomics use cases neuroglancer is built around, neuroglancer is far and away the best library I've used for looking at 3D scans in the browser. There is an email-gated demo here if you'd like to take a look at how we are using it.

Because my requirements diverge quite a bit, I have forked neuroglancer to make it "headless": I keep the WebGL rendering, viewer/state internals, chunk loading, and camera controls, but strip out essentially all of the built-in UI chrome. I then use a host application to drive everything, which supplies its own UI elements. I do this by:

• importing individual modules via the ./unstable/* export map rather than booting the standalone app
• maintaining a small fork where I expose an imperative control API over the Viewer class
• wiring that API into our host UI

CommandCatalog

The PR #1072 introduces a CommandCatalog class, which is an introspective enumeration of everything the viewer can currently do, kept in sync via signals. That class is very useful from an embed/host application point of view, and does much of the work my fork currently does (maintaining a list of "what can the viewer do right now" so that I can build UI elements and control the viewer).

There are a few other forks/wrappers of neuroglancer which I think would benefit from being able to use the CommandCatalog:

• react-neuroglancer wraps a forked version, that uses a props/callback control API. Theoretically this package could consume the CommandCatalog instead.
• ng-extend which is a Vue app that embeds neuroglancer as a git submodule

There are also a few issue threads in the same space: #326 #427 #439 and #731

Proposal / Discussion

There are two parts to this: a small near-term proposeal that's already raised as comments on the PR, and a larger direction I'd like to get feedback on.

Near-term: I've left a few suggestions on #1072 that would let my fork — and other consumers — use the CommandCatalog directly, by keeping it separable from the palette UI (a DOM-free module, a reachable catalog instance, and invocation lifted onto the catalog). None of these change behaviour; they just make the enumeration primitive usable without pulling in the UI aspects.

Larger: building on that, I'd propose a first-class way to register a command into the same catalog. My reasoning is that neuroglancer currently has two partial notions of "a thing the viewer can do", and neither is quite the right home for a host-defined command:

• Actions are lightweight and one-shot, but only discoverable via a keyboard binding — an action with no bound key is invisible to enumeration, which is why the catalog needs a hardcoded SUPPLEMENTAL_COMMANDS list to surface things like screenshot and edit-json-state. They also carry no metadata, so the palette reverse-engineers a label from the action-id (here).
• Tools are the opposite — enumerable independently of any key binding (via the tool-query mechanism, getMatchingTools) and self-describing via tool.description. A tool is stateful and layer-aware (e.g. an annotation tool), which is a lot of machinery for a plain "do X now" command.

There is a gap then for a labelled, enumerable command that a host can register, for which a keyboard binding is optional rather than the thing that makes it exist. A registerCommand function could fill this gap, and then the same mechanism would be available to host applications that embed neuroglancer. In my fork we have clipping planes, for example; registering that control might look like:

registerCommand({
  id: "gscan.clip-planes.toggle",   // stable + serialisable
  label: "Toggle clip planes",
  group: "Clip planes",
  isAvailable: () => { ... },        // observable / tracked
  invoke: () => { ... },             // parametric variants TBD
  defaultBinding: "shift+c",         // optional
});

The catalog would then be the union of registered commands + tools + built-in actions, and stay signal-driven — isAvailable would feed the same "what's usable right now" mechanism the palette already relies on. The palette renders that union as its UI; an embedder like me maps each command's invoke onto our own UI chrome.

Lastly, if this was all in place you could then completely split the UI concerns from the engine concerns internally, if all tools/actions/inputs were registered via the catalog. You could have a setupEngine() function which instantiates the rendering engine, mounts to DOM but doesn't create any UI chrome - this would go along with setupDefaultViewer which would remain unchanged. Applications which host neuroglancer could then use setupEngine() if they need to create their own UI, or use the default one instead.

Questions

• Is anyone else running neuroglancer as an embedded, chrome-less renderer? How are you handling it?
• Following on from that, how are you suppressing / replacing the UI?
• For maintainers: is making neuroglancer more consumable as a rendering + state + control-surface library something you'd see value in supporting?

[chrisj]

for me

is making neuroglancer more consumable as a rendering + state + control-surface library something you'd see value in supporting?
absolutely.

There is plenty interest in our side in being able to tailor the capabilities of Neuroglancer for specific workflows

[seankmartin]

I also agree. There may be some limits here and there. But overall at least considering these kind of things should only help. For e.g. I think the revised command system which you worked on and is largely is inspired by this need you have is better for any consumer, but also a stronger architecture for within neuroglancer itself