Improve multiple view/camera and Node parent<->child linking

[djhoese]

See #1666, #402, #1021, #1124 for more a lot of background on the issue.

Basically, the original started code of vispy included some cool demos of shared visuals/data between multiple views. There are also existing examples for sharing GL contexts. This is really what #1666 is talking about. The main problem is that eventually visuals had to be changed so they only had one parent. This simplified a lot of logic (I'm guessing), but limits the flexibility of a Visual. While it would be simple to just say "Oh now Nodes have parent and children properties" it won't actually work in all cases.

Some operations like transforms and in some cases determining the dpi of the main canvas are used at low-levels of the visuals/scene canvas. I'd like to really look into this and document where the issues are and what it would take to fix this.

[almarklein]

So Vispy currently has a scene graph in which each visual can have exactly one parent and this makes some use-cases hard, such as viewing the same scene in different canvases, or replicating partial scenes.

Some thoughts - I think there are (at least) 3 approaches to support such use-cases:

• Explicitly allowing multiple parents in the scene graph, as vispy had earlier. This makes things very complex and it breaks down in certain cases, as we've found.
• Making visuals not be aware of their parent (only of its own children). During events such as drawing, the chain of parents is available. This way a visual can be re-used as often as you like. I have implemented this in a (non-public) viz library. This worked pretty well, but there is no way to traverse the scene graph (except down the children) outside of such events.
• Allowing a single parent so that the scene-graph is well defined, but support sharing data and higher level components between scene graph elements. This is the approach that we took in pygfx (inspired by ThreeJS). Data for textures and buffers is stored in corresponding objects, and these can be shared. Further, each object in the scene has a geometry and material "sub-object", which can also be shared across scene graph elements.

To support the mentioned use-cases, we could try moving in either of these directions. I suspect going back to the first situation might not be a good idea. The second approach is probably difficult since we rely on being able to traverse the scene-graph (rightfully so). My gut feeling is that adding a mechanism to share data between visuals could be a good way forward, and will add other benefits as well. Bokeh has something similar (it calls it data-sources). In our case we can probably do with something like Buffer and Texture.

@Korijn maybe has thought/insights that I failed to mention

@rossant thoughts from POV of Datoviz and the new high-level API?

@campagnola as the person doing most of the work on the multi-parent implementation and later removing it, do you remember the reasoning?

[Korijn]

This is accurate. I would highly recommend sticking to lightweight scene graph objects that hold references to data such that the data can be shared between scene graphs (almar's third suggestion).

In essence this allows you to decouple rendering, scene construction and data management, while maintaining your scene's simple tree structure.

It's not only the simplest approach, it's also proven! All big industry standard rendering engines implement the scene graph abstraction this way :)

[djhoese]

@almarklein Nice write up. Based on this I think I agree with you on the multi-parent stuff. That was what I originally wanted to re-implement after reading old issues on multi-parent functionality being removed. This is mainly driven by some of the logic you can find sprinkled throughout vispy from "the olden days" but aren't heavily used in our every-day use cases. Things like "views" of Visuals (read-only interface to a Visual for access from other views) and the multiple camera views looking at the same visual like I mentioned in the original post above. I'm a little nervous about how things like the current TransformSystem works in your shared data in option 3.

That said, this shared data idea goes really well with a "data source" concept I've been mentioning in other places (gitter, gh issues, etc). I'd really like something like this for textures so we could do something like lazy loaded data backed by libraries like dask, or cupy, or even WMTS tiled image HTTP sources. My hope was that visuals could be made to accept these "data source" objects and using some defined interface could connect to various events/signals and request data in certain ways. This events/signals/callbacks idea works well for these data sources where vispy can be "alerted" that new data is available or has been updated and that the visualization needs to be redrawn.

For the data source objects you're talking about @almarklein, do they provide and manage the GL-like object in the current GL context? The Visuals would then just access that GL handle? Or something else? I assume this is the only way this can work for best performance?

[rossant]

@almarklein there's no scene graph in Datoviz at the moment, and I don't think there will ever be one. Datoviz will remain a relatively low-level library, and higher-level features should be implemented in external third-party libraries such as VisPy.

Something has always been unclear to me: what are the problems that the scene graph is supposed to solve in scientific visualization? Does anyone have a clear real-world scientific use-case where a scene graph is useful? What are the alternative solutions?

[almarklein]

Does anyone have a clear real-world scientific use-case where a scene graph is useful?

The obvious use-cases are simulation of physical bodies where the transform of an object also applies to its children (and their children). But also e.g. cases where you want to group a bunch of visuals and e.g. scale or translate them together. Adding a special visual that draws a box wireframe corresponding to the bounds of its parent. These are all things that can be solved without a scene-graph, but a scenegraph makes it a lot easier.

there's no scene graph in Datoviz at the moment, and I don't think there will ever be one.

I think Datoviz not having a scene-graph is not a problem, as it could be implemented in the new API, or in the Dataviz wrapper for the new API. What's more relevant is the ability to share data between visuals. Because that's about impossible to add in higher layers ...

One example for data sharing: showing multiple representations of the same data, e.g. imagine 4 panels where 3 show a slice through the volume and one shows a volume render. Would be nice if all three sample from the same GPU texture.

I'd really like something like this for textures so we could do something like lazy loaded data backed by libraries like dask, or cupy, or even WMTS tiled image HTTP sources.

Good point, this is also a role that such data sources can cover.

do they provide and manage the GL-like object in the current GL context? The Visuals would then just access that GL handle? Or something else?

Basically, the part of the VolumeVisual that handles the uploading of the data should be refactored out into a Texture class, and any visual that want to access that data on the GPU will thus use the corresponding handle.

[Korijn]

You don't have to use the nesting capabilities of a scene graph. If you only add children to the top node, it's just a flat list of visuals. If that's what you prefer, that is perfectly fine, and you can work with just that!

One example for data sharing: showing multiple representations of the same data, e.g. imagine 4 panels where 3 show a slice through the volume and one shows a volume render. Would be nice if all three sample from the same GPU texture.

As an example, implementing this with a scene graph would also remove the need for movement tracking glue code; you can add Camera nodes as children of the volume (or its slices depending on how you choose to implement this), so that when you move the slice, the cameras automatically follow because they are oriented relative to the slices.

[rossant]

The obvious use-cases are simulation of physical bodies where the transform of an object also applies to its children (and their children).

In this example, how many objects would there be, typically? Would they be instances of a "mesh" visual, or something else?

Would you assume that the final positions of all points are computed on the CPU or on the GPU? If you don't have a scene graph, am I correct that you could compute all positions on the CPU and updating them regularly? Or do you expect GPU acceleration for the transformations?

But also e.g. cases where you want to group a bunch of visuals and e.g. scale or translate them together.

I can see how this use-case would be useful in, e.g., a drawing application. But in a scientific application, I would imagine this would mostly concern "small" objects such as shapes (for example, drawing a selection box in a raster plot), in which case computing the transformations on the CPU would be fine. When it comes to large visuals with many points, I would say the most typical type of transformation is panzoom/arcball/non-linear transform which is specific to scientific applications, and can be supported directly without a scene graph.

Adding a special visual that draws a box wireframe corresponding to the bounds of its parent.

For this, I would just do a data.min(axis=0), data.max(axis=0) and add a wireframe cube visual with these points. Yes, the min and max are computed on the CPU, but I think that would also be the case with a scene graph. If you'd want to use the GPU, you could use a separate compute shader.

What's more relevant is the ability to share data between visuals. Because that's about impossible to add in higher layers ...

I agree. In Datoviz, visuals can easily share GPU buffers and textures. A visual can decide to own its data (typically, its vertex shader which is rather visual-specific), or to use a separate GPU object that can be shared between visuals.

One example for data sharing: showing multiple representations of the same data, e.g. imagine 4 panels where 3 show a slice through the volume and one shows a volume render. Would be nice if all three sample from the same GPU texture.

Sure, I have a prototype doing exactly this.

I think the reason why you are keen in having a scene graph, and I'm not, is that we have different target use-cases in mind. I target use-cases where you have relatively static data, few visuals with many points. It seems you're more interested in 3D applications with a relatively large number of dependent meshes with positions that depend on each other.

[Korijn]

relatively static data, few visuals with many points

Could you explain why you think a scene graph would not be suitable for this use-case? It should be as simple as adding a couple of data sources and visuals to a scene, and rendering, right?

[rossant]

Could you explain why you think a scene graph would not be suitable for this use-case? It should be as simple as adding a couple of data sources and visuals to a scene, and rendering, right?

I guess there's a kind of "proto scene graph" in Datoviz, it's very limited but for now it seems to be sufficient.

Here's how it works:

• You add a visual to the scene, for example a "marker" visual
• You pass the position of all points as an (N, 3) NumPy array, in the original data coordinate system
• The scene determines the appropriate transformation into a normalized coordinate system spanning [-1, +1]^3. This depends on all visuals in the scene.
• The data points are transformed once on the CPU.
• The visual has a chance to make further transformations on these points before uploading them to the GPU (for example, visual-specific triangulation).
• Dynamic interactivity is implemented on the GPU with model-view-projection matrices, or with non-linear transforms (not done yet)

It's mostly the hierarchical aspect of the scene graph that I feel is overkill in the type of use-cases I have considered so far, but maybe that's an oversight from my end.

[almarklein]

I guess you could say that not having a scene-graph is the same as having a scene graph which is only one level deep. And I agree that most use-cases would not need more than that. In fact, you're right that any use-case can be implemented without deeper nesting. It's just that a scene-graph provides a simple mechanism that makes implementing such use-cases much easier.

Not having a scene-graph (for a higher-level API) does not makes things faster. Maybe it makes the code a bit simpler, but not much ... there is a bit of recursion as you iterate over the elements of a scene, and you need to compose model matrices, but that's about it. More than worth it IMO :)

[rossant]

OK I see. So I guess fundamental questions are:

• Is a pure CPU scene graph sufficient? Or is there a need for some GPU support? If so, how would that work?
• What are the features you'd need from the low-level backend (such as Datoviz) in order to implement the scene graph of your dreams? If you're only interested in a pure CPU scene graph, I guess you don't need anything from the low-level backend beyond the ability of specifying raw positions (without the backend doing undesirable automatic transformations). The hard part is if you need some sort of GPU support.

[almarklein]

Is a pure CPU scene graph sufficient?

For by far the most use-cases yes. The easiest way to implement this might be to pre-multiply the model matrix with that of the parent before uploading it to the GPU. This could become slow for a huge amount of objects, but then ppl can resort to e.g. custom visuals.

BTW: In Vispy the transform and shader system are quite sophisticated. AFAIK each transform is done separately in the shader (so the level of nesting might affect the shader code).

What are the features you'd need from the low-level backend (such as Datoviz) in order to implement the scene graph

A way to specify the transform from model positions to world/scene positions.

[rossant]

OK so this might be much simpler than I thought. What I had in mind was the VisPy system, which I've always found too complex.

A way to specify the transform from model positions to world/scene positions.

A way to specify the vertex coordinates in the model coordinate system, + a way to specify custom MVP matrices (the transformation being implemented on the GPU) would be enough, correct? If so, you pretty much have everything you need in Datoviz.

Let's take the example where you have 100 meshes with interdependent positions that are updated at every frame according to some custom logic implemented on the CPU (which uses a CPU-based scene graph under the hood). You could create a single "mesh" visual in Datoviz. The library allows you to update the vertex positions of all points across all of your meshes at once (it is recommanded to batch multiple objects in a single visual as much as you can). You can also control the MVP matrices. Then I think you have everything you need!

[rossant]

Let's take the example where you have 100 meshes with interdependent positions

It makes me think that it might be worth using another terminology for these multiple objects being rendered with a single "low-level" batch visual. They could be called "instances" or "objects" or whatever, but not "visual": the point is that you can batch multiple objects within the same visual, which is good for performance. The CPU-based scene graph would not handle "visuals" but "instances" instead, and would be a totally separate system that the low-level backend knows nothing about.

[djhoese]

I'm not sure I have as complete an understanding of the implementation details as you guys do, but since @rossant mentioned it, what is "CPU" about a CPU-based scene graph? Isn't VisPy's current scene graph a CPU scene graph? You mentioned in Datoviz that data points are transformed once on the CPU. Does this CPU-based scene graph (to be done in VisPy) have to do this?

For me, and I think this is what you two were saying, the biggest thing with the scene graph in vispy that would require something from the lower-level graphics library would be the ability to "inject" new shader code. Specifically this is needed for transforms. I use this type of hierarchy in one of my own projects to have a map of Earth in one project (one graph node) then nodes for any datasets in different projections (each their own node) and their children are the actual visuals with their projected vertices that get transformed on the GPU.

The only other/new feature from VisPy's point of view is this ability to share GPU elements (buffers, textures, etc).

This case of 100 meshes with interdependent positions comes up a lot in vispy. Users assume that because they can have multiple Visuals and because VisPy has visuals like "Sphere" that they should just make individual Sphere visuals and VisPy can just handle 10s or 100s of these easily. The traversal through these and the serial drawing by the GPU makes it get slow quickly. You also have cases where users design their application to create new Visual instances when what they should really do is update the data in an existing one (ex. "I got a new image, let's stop showing the old ImageVisual and create a new one").

As for naming, this isn't a good term for what you're talking about but I've been watching Loki on Disney+ so "variant". We could go as lame as "component" or "sub-component" of the Visual. Or we could go with "subset" but that doesn't seem right either. Or "segment" as in one line segment of many lines.

Edit: Coworker just brought up "VisualFragment"

[rossant]

what is "CPU" about a CPU-based scene graph?

I mean that the transformations of the points are all done on the CPU.

Isn't VisPy's current scene graph a CPU scene graph?

I think it supports GPU-based transforms.

Specifically this is needed for transforms. I use this type of hierarchy in one of my own projects to have a map of Earth in one project (one graph node) then nodes for any datasets in different projections (each their own node) and their children are the actual visuals with their projected vertices that get transformed on the GPU.

Could you make a demo of this at the next meeting? I'd love to see this in action, it would help me think about ways of implementing it. Datoviz does not support shader code injection at the moment, and it would be hard to implement it in the C library (but a bit easier to do in the Python wrapper). I wonder if we can support a use-case such as yours without shader injection.

This case of 100 meshes with interdependent positions comes up a lot in vispy. Users assume that because they can have multiple Visuals and because VisPy has visuals like "Sphere" that they should just make individual Sphere visuals and VisPy can just handle 10s or 100s of these easily. The traversal through these and the serial drawing by the GPU makes it get slow quickly. You also have cases where users design their application to create new Visual instances when what they should really do is update the data in an existing one (ex. "I got a new image, let's stop showing the old ImageVisual and create a new one").

This is an important point. Something that can be partly tackled with good examples (which clearly show what are good and bad practices) and documentation.

As for naming, this isn't a good term for what you're talking about but I've been watching Loki on Disney+ so "variant". We could go as lame as "component" or "sub-component" of the Visual. Or we could go with "subset" but that doesn't seem right either. Or "segment" as in one line segment of many lines.

Propositions (thank you thesaurus):

• component
• subset
• segment
• fragment (but clashes with fragments in the fragment shader)
• bit
• chunk
• piece
• portion
• atom
• bite
• fraction
• slice (corresponds to a NumPy slice such as vertices[n:m, :])
• hunk
• quantum
• parcel

[almarklein]

I mean that the transformations of the points are all done on the CPU.

Why would you do that? Why not a simple matrix multiplication on the GPU? Thinking e.g. of applications where the data is frequently updated.

[rossant]

Sorry, I didn't write what I meant to say. :)

In Datoviz, there are two transformation stages :

• the CPU transforms, that go from the original data coordinate system to the normalized data coordinate system on the GPU (these correspond to the GLSL attributes in the vertex shader)
• the GPU transforms, which are just model-view-projection right now, performed in the vertex shader. The output is in gl_Position.

There's no explicit scene graph, but it's equivalent to one.

The reasons for this separation of concerns:

• I need double precision for the original data normalization and transformation (it could also encompass triangulation)
• In many use cases, the data isn't updated very frequently (compared to the frame rate) and we can afford doing the transformation on the CPU
• For MVP transforms, single precision is mostly sufficient

The next implementation step will be to add nonlinear transforms on the GPU: they might not require shader injection if we have a predefined set of commonly-used nonlinear transforms (e.g. Earth projections).

[djhoese]

Could you make a demo of this at the next meeting?

I've added it to the agenda. Not sure I'll have time to give a really good demo, but I hope to work on it at some point today.