Chances for Three.js interop/wrapper?

[jussty]

What a great project!

Since this is a lot more low-level than NGL, how are the chances that there will be an official interface to Three.js at some point? It would make sense to include molecule geometry into existing tools, given the huge ecosystem. E.g. implementing advanced controls for XR with raycasting, colliders etc. will be hard (and unnecessary) to do from scratch for this project.

Mol* seems to move fast paced at the moment, so maintaining a wrapper may currently be a liability. Could you give a an estimate when in the future will be a good point that such a thing may be implemented?

(I am coming over from a related issue with NGL)

[dsehnal]

I don't think we will ever have an official interface for Three.js.

However, we have a sub-project planned for generating library-independent geometries. These could then be imported in Three.js and other solutions.

We have a theming refactor planned for the mol-repr module soon and after that I think the code will be stable enough to write the exporter. Maybe you want to help with that?

[awiddess-ccdc]

@dsehnal is it worth spending a little time on the sub-project to support the library-independent geometries then allow this project to utilise the api?

[arose]

is it worth spending a little time on the sub-project to support the library-independent geometries then allow this project to utilise the api?

the current low-level work we are doing is to create an exporter interface for RenderObjects (create as part of representations and containing all geometry and coloring data) to transform them into something useful elsewhere. The first exporter will be a OBJ exporter but once we have a stable exporter interface exporters for THREE.js geometries or other formats can be added.

We want to use it for the Viewer app to offer download of OBJ files for the whole scene. But one can also load structures, create representations using low-level mol* functions and then pipe the representations' renderobjects through an exporter programmatically.

[jussty]

the current low-level work we are doing is to create an exporter interface for RenderObjects (create as part of representations and containing all geometry and coloring data) to transform them into something useful elsewhere.

That sounds great, should be easy to derive Three.js geometries off this data. This is a very good approach, since it allows to ship geometry loaders without the need to carry whole Web Applications around as "library".

The first exporter will be a OBJ exporter but once we have a stable exporter interface exporters for THREE.js geometries or other formats can be added.

It might make sense to leverage some of the cooler features of the Viewer/mol-gl code (e.g. shader based all-atom view for big scenes) for Three.js exports too, since a simple all-atom geometry of a huge protein complex consisting of ordinary spheres will not be very useful in Three.js for performance reasons. But of course I don't need to tell you that ;)

We have a theming refactor planned for the mol-repr module soon and after that I think the code will be stable enough to write the exporter. Maybe you want to help with that?

@dsehnal Thank you for the invitation, I will do my best to keep up with the project. This is mainly hobby/personal fulfillment for me and my (unrelated) day job startup will take its toll during the next months (market entry), so I don't want to promise to contribute anything meaningful.
I consider porting my pet project from its current platform to a mol-plugin though, so maybe the insight gained from this proves useful.

[dsehnal]

@dsehnal is it worth spending a little time on the sub-project to support the library-independent geometries then allow this project to utilise the api?

@awiddess-ccdc I suppose I could write some basic mesh abstraction that would be able to represent a good deal of what we do in a manner that could be consumed by other code.... and then someone else can work on top of that. I will think about how difficult that would be.

[awiddess-ccdc]

This sounds great, probably something I can help with?

[dsehnal]

We want to have a server for providing geometries "on demand" (akin to Volume- and ModelServer currently available). I.e. you provide PDB id (or upload a structure) and you get a geometry back.

It would be a nice standalone project within Mol*. Would you be interested in doing that?

[awiddess-ccdc]

Absolutely, and sounds useful for the project without impacting the architecture.

[dsehnal]

Alright, here's something to get you started:

• The molrender project shows how to construct a scene without the "plugin".
• The scene has "renderables" array from which you can extract the data to be rendered.
  • Here is the entry point for the renderable data.
• You can use volume or model server as a basis for the API (it just uses nodejs express)
• Just add the server to src/servers/geometry.
  • No extra stuff to compile is required, npm run watch-tsc will do for quick recompiles.
  • It will compile to lib/servers/geometry from which you can run it for development.

The geometry itself should be quite easy to export, what might be more problematic is correctly handling the coloring. Perhaps you can figure that out yourself :) Also, there is a lot of instancing involved, especially for the assemblies so finding a suitable export format that supports that would be nice too.

If you have any more questions, please do ask.

[awiddess-ccdc]

@dsehnal I've gone through the geometry and that of Threejs as well.
If you want this as a separate standalone service do we create a separate repo for it?
(if so, will need a repo created).

Are you happy with an IoC approach for request geometry based on what's passed in?

[dsehnal]

If you agree, I think it could be part of the main repo same as model and volume servers are as I mentioned above:

• src/servers/geometry for the web API
• src/mol-geo/export (or perhaps src/mol-repr/export depending on where it fits better) for the exporters.
  • The reasons for this is so that these are separate and could be easily used from the plugin.

The way you architect it is up to you. I would recommend defining some intermediate representation for the mesh data that models things like instancing and coloring and then writing exporters/converters based on this intermediate representation.

Something like

interface ExportableMesh {
  ...
}

interface MeshExporter<P = any> {
  name: string, 
  descriptoin: string // For UI diplay,
  export(mesh: ExportableMesh, builder: StringBuilder | BinaryBuilder, params: Params): Task<void>
  // doesn't need to look like this
  // there is StringBuilder naemspace in mol-util/string-builder.ts
  // suppose something similar could/should be written for Binary data as well
  // Task is from mol-task module, allows to track progress, it's used all over molstar
}

namespace MeshExporter {
  export type Params<E extends MeshExporter> = E extends MeshExporter<infer P> ? P : never
}

const MeshExporters = {
  'mesh': MeshExporter,
  'stl': StlExporter,
   ...
}
type MeshExporters = typeof MeshExporters
const MeshExporterName = keyof MeshExporters

// with the MeshExporter type, you can have code completion for function like
function exportRepresentation<E extends MeshExporterName>(
  repr: Representation,
  format: MeshExporterName, params?: MeshExporter.Params<MeshExporters[E]>) {
  // 
}

// we use this approach in src/mol-plugin-state/builders and it works quite well
// there are also "registries" as well to make it extentable on the fly if you so choose

[arose]

Hi @awiddess-ccdc, I agree with what @dsehnal suggested above. Just some details.

Instead of MeshExporters it would be nice to have GeometryExporters as many formats support non-mesh data like lines, points or even spheres.

I would like that the actual exporting of the geometry data is done on the level of RenderObjects. To large extend the geometry exporting can be seen as a kind of 'rendering' and that way it would work independently of Representation objects. I have a student (Jesse, @dudeyoudontknow) who is working on an .obj exporter using that approach. There is a branch with some stub interfaces (https://github.com/molstar/molstar/tree/renderobject-exporter).

In case you want to include 'molecular data' (like atom names) into the exported data it is good to have a higher level interface around the renderobject exporters that works on Representation objects and also gives easy access to molecular data via the Representation.getLoci method.

[arose]

Did anyone work on a Mesh or Geometry exporter?