nannou_ui - A more efficient, nannou-friendly approach to UI compatible with future GUI editor plans.

[mitchmindtree]

This issue is an actionable follow-up to the conclusions drawn in #2. Here I'll track my progress and plans, updating this top-level comment as I go.

Much of the work will draw from my work on conrod and the lessons learned during that process. That said, nannou_gui will differ in some fundamental ways:

• nannou_gui will remain tightly associated with nannou, leveraging its existing color, geometry, mesh and windowing abstractions. This will solve some annoying, existing type compatibility issues that currently exist in nannou's Ui, such as the need to use different color types between the UI and Draw APIs.
• nannou_gui will be based on a "retained-mode" API rather than an "immediate-mode" API. There are a few reasons for this:
  • Performance. The existing immediate mode API is more than efficient enough for projects requiring small, non-complex GUIs, but can become quite taxing on CPU in large, sophisticated applications (see spatial_audio_server). This is due to the requirement for updating all visible widgets once every update, regardless of whether or not they have changed. That said, the ergonomics provided by immediate mode GUIs are highly beneficial. It should be possible to implement an immediate-mode API over the retained-mode API with an approach involving a lot of caching, which is largely how any slightly-efficient immediate mode GUI works internally anyway.
  • GUI Editor. nannou_gui is designed with GUI editor usage in mind - that is, the ability to design and develop efficient GUIs via a GUI. This requires the ability to treat widgets and their events as serializable data, an approach that is not currently possible with the existing immediate mode API in an efficient manner.

Blocking issues

• Add text logic module and support for drawing text via the Draw api #77 Add a text module to nannou for basic handling of fonts and layout.

Roadmap

• Create repository.
• Settle on design for state synchronisation.
  • Research existing native-target, Rust UI efforts and discuss in comments below:
    • conrod (immediate)
    • druid (data-driven)
    • orbtk (reactive)
    • iced (reactive)
    • azul (immediate)
    • moxie (declarative)
    • reclutch (retained toolkit)
    • makepad (software dev platform)
    • Consider "classic" retained GUI.
• Graph - The data structure describing widget ownership and layout.
• Events - UI event types and functions for interpreting them from raw windowing events.
• Widget - a trait implemented for all widget types. Either this trait, or a closely related SerdeWidget trait should require Deserialize and Serialize and allow for trait object serialization via the typetag crate. While designing this interface, it will be important to consider that it should be possible to create widgets and their implementations entirely from scratch at run-time. Edit: alternatively, consider storing dynamically created widgets as code?
• Theme - a flexible data-structure allowing for the association of unique styling defaults per widget.
• Primitive Widgets - a small collection of widgets from which all other widgets may be composed:
  • Mesh - indexable triangulation consisting of a collection of vertex attribute channels. E.g. position, texture coordinates, colour, normals.
  • Text - based on the rusttype crate.
  • Crop - crops children to rectangular area, maps directly to wgpu scissor implementation.
  • Path - lyon path that may be stroke/fill tessellated.
  • Blend - all children have the specified blend mode applied.
  • Layout widgets (inspired by druid/flutter):
    • Pad - pads area for specified child.
    • Flex - container automatically distributing children along a range.
    • Flow - container automatically distributing children along multiple ranges (row by row or column by column).
    • Split - container for two children, split in the middle.
    • Either - container conditionally showing child A or child B.
    • Fixed - child has area with fixed size and position (aka not responsive to flex).
• Render pass for UI.
• DynWidget (or DataWidget or DynamicWidget) - a highly flexible, serializable Widget implementation allowing for arbitrary configurations of children widgets at runtime. This will serve as the base type for widgets composed within the GUI editor.
• Proof of concept examples:
  • Demonstration of each primitive widget.
  • Simple graph widget + gantz demonstration.

Upon completing this roadmap, nannou_gui should be ready to serve as the basis of a nannou_gui_editor crate.

[mitchmindtree]

Synchronising GUI and Application State

One design choice I've yet to resolve is how to handle the updating and synchronisation of state in general. This can be broken into two related problems:

1. Allow changes in application state to update GUI state.
2. Allow changes in GUI state to update application state.

In conrod this is generally handled automatically as widgets require being instantiated directly from application data on every update. The trade-off here is the performance issue of needing to update every widget every update, a large cost that we are trying to avoid in the new design.

Solution Research

Data-Driven (ala druid)

The druid library uses a data-driven approach not dissimilar to conrod where widgets are updated with a reference to the application state itself. It provides similar benefits, making it trivial to keep the GUI synchronised with application state and allowing the GUI to store slightly less state itself.

However, the druid API is different in the sense that it provides a reference to the previous application state as well. Druid also requires that application state implements Data, a druid trait that is almost identical to PartialEq, allowing to compare whether or not the previous state is equal to the current state. This provides an easy, cheap approach for determining whether or not Widget::update needs to be called for each widget each frame.

State updating occurs via a couple of methods:

1. druid::Widget::update for updating the widget in response to some change in application state.
2. druid::Widget::event for updating both the widget and application state in response to some application/window/user-input event.

One of the tricky requirements of the druid approach is the requirement for the Data and Clone implementations on all application state that must be visited by the GUI. This means even common standard collection types like Strings, Vecs and HashMaps can be expensive to maintain, and that in general the user would benefit by understanding how to structure their application with immutable data structures (e.g. the im crate).

Another requirement to consider is that widget trait implementations are templated on the type of data that the widgets are compatible with. This could make our goal of widget serialization a bit trickier to achieve as the typetag crate (necessary for enabling the serialization of trait objects) does not support generic traits. This could possibly be worked around by using a separate trait for widget serialization, e.g. SerdeWidget, that is only implemented for widget implementations over some dynamic data representation.

Unfortunately, we can't try using druid directly in a nannou app for a couple of reasons:

1. druid rolls its own shell (windowing library) from scratch. This means we can't just convert winit events into some druid input event type to drive it forward - druid itself requires managing windowing, events etc. This makes it impossible to run alongside nannou, as both require ownership over the main event loop, of which there can only be one, particularly as the event loop requires running on the main thread on some platforms.
2. druid requires using the piet 2d rendering library for rendering its graphics. piet does support multiple backends, but no wgpu backend just yet. I imagine even if piet does land wgpu support, it might be confusing to users to have to switch between entirely different renderers going between the draw and ui APIs.

We would also begin to run into some of the same issues as we currently have with conrod where users cannot use nannou positioning and colour types direclty in the UI API or vice versa, requiring a conversion step between the two which is frequently confusing for new users.

Manual Synchronisation ("classic" retained GUI)

On the opposite end of the spectrum, we could opt for a more classic retained approach, where the GUI remains entirely separate from the application state and does not require a reference to it during rendering. In these designs, updating the application state can be achieved by either callbacks, channels or matching on widget IDs alongside their associated events. Synchronising GUI state can be achieved by manually indexing into the gui with widget IDs and updating state as necessary.

The tedium of the manual approach is often related to the separation of all of the steps at which synchronisation is necessary. E.g.

1. On intialisation, the GUI should be initialised with state that reflects the application.
2. On user input, windowing and application events, GUI interaction should be able to update application state.
3. When application state is updated via other means (e.g. I/O, OSC, audio/video, controller, etc), the GUI must be updated to match.

Conrod's immediate design consolidates all of these steps into a single step using caching to hide the distinction between initialisation and repeated updates. Druid's data-driven design is similar, with the slight difference that steps 2 and 3 are split into two methods. While these two libraries provide arguably simpler APIs, both do so at some other cost (performance in the case of conrod, strict Data and Clone requirements on application state for druid).

The major benefits of the manual synchronisation approach seem to be performance and API flexibility. No work is performed except when absolutely necessary, and no restrictive trait implementations or design constraints are required to achieve this. It should also be easy enough to build an immediate or data-driven API on top of a more classical retained API, whereas it would make less sense to do the reverse due to the performance cost that has already been paid for the immediate design, or the state restrictions that are already imposed in the data-driven design.

That said, another issue with the classic retained approach is that widgets need to independently store all necessary state for rendering and handling events. E.g. in order for a list to present unique text for each button, it must store its own container with a String for each element. On the other hand, a data-driven design has the benefit of referencing that data directly from the application state during rendering or event handling. That said, the data-driven approach needs two copies of the whole application state when updating widgets anyway, so it could be argued that it's not necessarily more memory efficient unless taking care to structure application state with immutable data structures. Then again, immutable data structures are renown for making it easy to cause unintended leaks in other ways due to the amount of reference counting required.

To be continued...

[Type1J]

My 2 cents: The more dynamic the UI is, the more the immediate seems appealing. Even in less dynamic UIs, if the UI isn't being changed, the render of the UI could be cached.

I'm eager to see how nannou_ui turns out!

[qingxiang-jia]

(I am new to Nannou so my question could be completely off.) It seems we are going implement our own UI rather than relying on conrod. If that's the case, are we going to use draw::Draw for drawing widgets? If so, wouldn't it be inefficient regardless the immediate mode or classic retain approach?

My understanding is, once to_frame is called, all drawing commands will be send to GPU for rendering. Unless we have a separate Draw for each widget (The doc says we can), to_frame will render everything again anyways. But if we only want to paint the widget that has changed, then we potentially will have many instances of Draw, which could potentially consume too much memory.

[Type1J]

If a UI renderer is drawing individual widgets at a time, then it will be very slow because GPU state transitions will happen much more often than needed.

If you look at a modern UI renderer, like QML from Qt (not viable here due to GPL), drawing doesn't happen at the widget level. Widgets are useful abstractions for the programmer, but the tree of widgets needs to be translated into a render data structure before drawing. The render data for a single draw contains data from multiple widgets.

Also, after looking at the speed of some web frameworks like React and Svelte (Svelte is faster) and also game engine UIs (almost exclusivly retained except for IMGUI), I'm leaning toward retained mode with signals at this point. It has less need for CPU when it's not doing anything, and, more importantly, it allocates much less often.

(edit: typo)

[qingxiang-jia]

@Type1J , thank you for the information. Indeed I don't know anything about UI libraries and hope I could improve. Do you have any suggestions for learning materials?

[Type1J]

Sure. The transition to using the GPU for UI happened around the late 2000s to the mid 2010s. Here's Bea Lam's presentation talking about rendering differences between Qt Quick 1.1 to Qt Quick 2.0. (Qt Quick is the actual UI framework for QML in Qt. QML is really just a declarative language to control the rendering engine.) I had to dig this YouTube link up, since it was such a long time ago, but this should give you a good overview of how and why we changed from drawing UIs in the way that you described before (Yes, they were drawn that way at one time.), and the way that we draw them, now.

https://youtu.be/iSDnjKvugcQ?t=355

[qingxiang-jia]

I really appreciate the help. I will watch the video tonight (also I am sorry for straying it away from its original topic, I will stop here).