Compatibility with FRP event handling

[milibopp]

As I mentioned in this blog post (also nice for motivation and background, in case anybody is unaware of what I'm trying to do), it would be pretty cool, if conrod's UI could be used in conjunction with FRP event logic. Now I assume that conrod does a lot of (imperative) event handling logic on its own. I'm not particularly familiar with its internals though.

I wonder if it is possible to find some common ground between a functional approach and what conrod currently implements. Possibly at least the rendering part could be shared, though I'm afraid most of the event handling logic cannot be shared.

In any case, I would be glad about your thoughts regarding whether it makes sense to try to use parts of conrod in this way and how it could be done. There are two concrete questions:

• Is conrod's rendering of UI elements decoupled from its event handling logic?
• How do you see the relation between UI rendering, conrod's current event handling and a future functional reactive event handling?

[mitchmindtree]

@aepsil0n I love where carboxyl is going, keep it up! I think conrod's possibly already quite compatible? Although it's quite imperative internally, it exposes a very functionalish API.

Re decoupling of rendering from event logic - we just landed this in the same PR in which elmesque was integrated :)

All event handling, widget state updating, mutation and reactions are handled within the .set(UiId, &mut Ui) method. During this stage, the widget builds their Element and using their unique identifier (UiId) stores them in the Ui's widget_cache (along side their state) ready for rendering.

Rendering of all the widgets occurs in a single step when the user calls ui.draw(graphics_backend), where all elements are collected from the widget_cache, sorted by depth and capturing, and finally rendered.

From a functional API perspective each widget has a couple inputs and outputs:

• Inputs are:
  • User input events like keys / mouse / window resizing (this is supplied via the Ui).
  • The previous widget state (also supplied via the Ui and UiId).
  • Some app data like string for a textbox, bool for a toggle.
  • A conditionally triggered function supplying the new app data state via the args (in the form of the .react(closure)).
• Outputs are:
  • The reaction.
  • A graphical element for rendering (stored straight in the Ui).

Perhaps it would be possible to make a wrapper function for a widget like this:

fn ui(input: Stream<(&mut AppData, &mut Ui)>) -> Stream<(&mut AppData, &mut Ui)> {
    input.map(|(app_data, ui)| {

        Toggle::new(app_data.switch)
            .react(|new_switch| app_data.switch = new_switch)
            .set(SWITCH, ui);

        // Other widgets...

        (app_data, ui)
    })
}

Hmmm I'm really just throwing around ideas, what are your thoughts? What would you like to see?

[milibopp]

Unfortunately it won't work just like that. One fundamental aspect of FRP is, that it abstracts away mutable state. Hence, we can't just put a &mut T into a stream. The trait bounds on the type in a stream even forbid this, as everything must be Send + Sync + Clone. Well, you might get it to "work" using Arc<Mutex<T>> instead, but the result would be highly undefined behaviour as Carboxyl expects the functions passed to it to be free of side-effects. Rust's type system simply does not allow to express this expectation.

And this is the reason why we probably can't reuse a lot of conrod's existing event handling system, because it appears to rely on mutable state. But anyway I will try to build a simple checkbox with carboxyl and elmesque to demonstrate how the state would be handled in a case like that. Then you can probably judge better, in how far conrod can be made conform to this pattern. In any case, it is already really helpful to have a functional API to create an Element from a button.

The more I think about it, it appears to me that FRP actually solves pretty much the same problem as conrod's Ui (without the drawing part). The callbacks you register in .react(…) calls are replaced by building streams and cells declaratively. And with FRP you also generate the Elements only upon state changes. btw does conrod's widget cache do anything more than that (like caching past states that might occur again)?

What could (theoretically) be reused from conrod's event system are the purely functional parts that look like this:

/// Map a widget's current state to its new state given an event
fn update(old: Widget, event: Event) -> Widget;
/// Does a mouse click hit a button?
fn hits(button: Button, position: (f64, f64)) -> bool;

[mitchmindtree]

As conrod is currently, widgets are quite heavily dependent upon the Ui for a number of reasons - most of them should be listed here. I'd be really interested to see a draft or outline on how FRP could solve each of the tasks listed there!

[milibopp]

That's good to know. Let me address these in general terms first. I'm working on a more concrete example at the moment.

Contains the state of all widgets which can be indexed via their UiId

The behaviour of a widget can be described as a Cell<WidgetState>. Cell is a generic type to describe some value that changes over time. One can use .sample() to access its current value. It probably requires some thought about how to manage the widget state efficiently, as in-place mutations of a contiguous Vec of widget states should be avoided.

Stores rendering state for each widget until the end of each render cycle

In the current examples, the content of the window is assumed to be one big Element. Drawing is done imperatively, see here. I am curious what kind of draw state must be stored there, that is not handled by graphics or elmesque.

As a side note, maybe it is a good idea to have some kind of Draw trait (impl'd by Element), when you e.g. want to combine conrod with some other graphics element. On the highest level you could then use trait objects to compose different implementations in an extendable way. But I'm unsure where the place for such a trait would be. Maybe @bvssvni or @Potpourri could weigh in on this?

Contains the theme used for default styling of the widgets

At some point you need a function view: WidgetState -> Element to map from a Cell<WidgetState> to a Cell<Element>. I suppose that such a function could be a closure over a theme, as long as the theme itself does not contain any mutable state. Lazy initialization should be okay here, as long as it is encapsulated and does not jeopardize referential transparency in any way. To make this more convenient, carboxyl must be able to deal with non-static lifetimes of types. For now, it would be easier if the theme could be owned by that function.

Maintains the latest user input state (for mouse and keyboard)
Maintains the latest window dimensions

The Cell<WidgetState> mentioned earlier will have to be constructed in terms of a Stream<ButtonEvent>, a Cell<(f64, f64)> and similar reactive components as provided by carboxyl_window. You explicitly declare these dependencies and the callback registration and updates are handled by carboxyl.

[mitchmindtree]

These all sound great!

While I think of it, there are a couple of other things aren't listed that the Ui takes care of:

Relative widget positioning. Conrod allows you to describe layout relatively i.e.

• .left(pixels)
• .down_from(other_widget_ui_id, pixels)
• .align_right()
  The Ui enables this by both storing the previously set widget's UiId and the positioning of all previous widgets.

Capturing. The Ui also keeps track of whether or not a Widget is currently capturing user input (i.e. an open DropDownList captures the mouse, an active TextBox captures the keyboard).

I am curious what kind of draw state must be stored there, that is not handled by graphics or elmesque.

The only draw state Conrod holds onto is an Element for each widget, and the depth of rendering (used to sort the widgets into order for rendering the elements).

maybe it is a good idea to have some kind of Draw trait (impl'd by Element), when you e.g. want to combine conrod with some other graphics element.

Can you elaborate on what you mean by "combine conrod with some other graphics element"? Do you mean giving access to its Element so that it can be collaged into another Element? Or something else?

[bvssvni]

@aepsil0n I liked your article!

[milibopp]

Yes, these two are more problematic. I just stumbled over it trying to refactor the toggle widget's set method.

Relative widget positioning. When I understand this correctly, it makes a difference for the layout, in what order set is called for widgets. This would work better, if it were more declarative like Elm's element API. But hey, you have already ported this to Rust, so I think this problem could be resolved by leveraging Elmesque to express these relations. This is probably quite a bit of work.

Capturing. I think this is essentially a part of event handling. It can be expressed using FRP primitives as well. You just have to filter the input stream for each element taking into account the entire UI. So if I picture the UI as a Cell<UI> (whatever type UI winds up to be), capturing e.g. keyboard input for a widget would look like this:

// keys: Stream<Key>
// ui: Cell<UI>
// widget_id: UiId
ui.snapshot(&keys)
    .filter_map(|(ui, key)|
        if ui.active_id() == widget_id { Some(key) }
        else { None }
    )

(ignoring the details for now)

Can you elaborate on what you mean by "combine conrod with some other graphics element"?

Yeah, sorry, that was a bit fuzzy. On one hand, I was talking about a collage with other Elements, which I think is covered by Elmesque, but conrod would have to expose it. However, I can also imagine a use case, where you want to combine this with some other library built on top of graphics. Looking even further, it would also be cool to render conrod UIs on textures embedded in some 3D application. But I'm probably getting ahead of myself with this. Let's focus on rendering Elements for now.

Conclusion

It looks like there's a lot to do. I think I have to prototype a couple of simple UI widgets out-of-tree to get a better idea of the precise API requirements. Anyway, generally it would be very beneficial to factor out the purely functional parts of the conrod internals.

[mitchmindtree]

@aepsil0n #424 is related (a draft for separating Widget trait implementations from the Ui and mutability entirely).

[milibopp]

So, as a quick update: what I did in milibopp/carboxyl#58 should pave the way to allow for some mutable state wrapped in FRP primitives. The essential idea is to provide a functional API around efficient in-place updates via &mut T. I have iterated on the initial design a little bit to provide better guarantees (I'd still like to test the equivalence to the existing purely functional API a little more though).

Nonetheless, this change allows one to integrate with conrod's current UI object, until #424 is resolved.

[milibopp]

I've been a bit absent from Rust development lately. But I'd like to give the integration of these two libraries a try once more. I see #424 has been merged. Does that imply that conrod's API can be used in a more functional manner now?

[miketang84]

also care the progress of this topic.

[milibopp]

I am working on some general considerations about reactive architecture with continuous time semantics, which might inform, how to approach this. But I can't really judge whether conrod is sufficiently immutable to allow an immediate integration yet.

One interesting note though: I've thrown out SignalMut, because I think it is a leaky abstraction and does not really help a lot. Maybe it would be feasible to write drivers for conrod as a separate library to interact with it using a purely functional API. By "driver" I mean some thread running in the background that samples signals, listens to streams and feeds into sinks ideally. Pretty similar to what I've done in carboxyl_window.

[miketang84]

hard work, but I notice that conrod now is reactive mode, but seems no frp. Maybe author of conrod can explain his design and plan.

[mitchmindtree]

I am working on some general considerations about reactive architecture with continuous time semantics, which might inform, how to approach this.

@aepsil0n definitely keep us in the loop! Whether or not this turns out suitable for conrod, I'd love to hear about your findings myself 😸

Maybe author of conrod can explain his design and plan.

Hmmm I haven't given too much thought to FRP compatibility since the last time we spoke on this - I haven't had an explicit goal in mind to steer conrod in that direction, but i'd absolutely be open to ideas about improving the API in whatever ways FRP can offer.

I think part of the reason I haven't given it a lot of thought is simply due to not having done any FRP in the last year (been pretty exclusively hacking in rust come to think of it!). I've generally been feeling out the design of conrod as I go, trying to find solutions to problems that fit rust's ownership system nicely. This is how the reactive-mode/retained hybrid came to be - the reactive API solves a lot of ownership issues for the user, while the retained internals (see the widget graph, which is pretty much a giant cache and widget relationship description) offer the performance benefits of keeping state around. There's a small bit about this in the guide.

Maybe it would be feasible to write drivers for conrod as a separate library to interact with it using a purely functional API. By "driver" I mean some thread running in the background that samples signals, listens to streams and feeds into sinks ideally.

This sounds pretty great! I'd be pretty interested in seeing how this works.

Another thing to note is that since the last time we spoke, conrod has stopped using elmesque for graphics in favour of adding a set of "primitive" graphics widgets. We found that using elmesque for graphics was often confusing for users, as it had a very different API to the rest of conrod. Now users can instantiate basic graphics the same way they'd instantiate other widgets. We can also cache graphics data in more clever ways now that graphics elements (now widgets) are a part of the same graph as the rest of the widgets. We also no longer require a draw method for Widgets, as their graphics can be created by instantiating child primitive widgets in the update.

These changes might make it more difficult to interop FRP with conrod, now that conrod no longer returns an Element describing its entire graphical state. However, it should definitely be possible to create a new type that plays a similar role to what Element used to, if need be.

I should mention; one of the reasons I didn't change conrod to follow a more elmesque style (as opposed to bringing the graphics closer to conrod's existing widget style) was due to the complexity of the relationships between widgets and the shape of the Graph. Conrod's widgets form a DAG, whereas elmesque's graphics, although nice to instantiate, quite strictly take the form of a rose tree data structure in its current state. I also anticipated that this rose tree, being entirely encapsulated within an enum hierarchy, would also have made it quite tricky to re-use and update parts of the tree. I thought about some ways to extend elmesque to support more of a DAG structure (perhaps I should have turned to carboxyl for this?), however most of the ways I could come up with would have involved a whole re-write. Conrod already had a widget DAG ready to go and it wasn't clear that the time involved in re-writing would have been worth the trade-off, which also made the graphics->widgets change a little more inviting.

Anyway, I'm interested to hear both your thoughts on all this!

[milibopp]

I think part of the reason I haven't given it a lot of thought is simply due to not having done any FRP in the last year (been pretty exclusively hacking in rust come to think of it!). I've generally been feeling out the design of conrod as I go, trying to find solutions to problems that fit rust's ownership system nicely. This is how the reactive-mode/retained hybrid came to be - the reactive API solves a lot of ownership issues for the user, while the retained internals (see the widget graph, which is pretty much a giant cache and widget relationship description) offer the performance benefits of keeping state around. There's a small bit about this in the guide.

Thanks for your update on this. I haven't been following conrod's development much, so this is really appreciated.

My general architecture approach strives towards a purely declarative API for users, which lets the library take this apart into all the imperative calls that need to be done to persist state, render stuff, etc. (this is what carboxyl and its dependents do).

Your idea sounds similar in some ways. I would also compare this to the virtual DOM approach taken in frontend architecture these days. And I believe it makes sense for us (i.e. Rust devs) to take a similar approach, even though we can skip a lot of stuff, because we don't need to work with a DOM.

Another thing to note is that since the last time we spoke, conrod has stopped using elmesque for graphics in favour of adding a set of "primitive" graphics widgets. We found that using elmesque for graphics was often confusing for users, as it had a very different API to the rest of conrod. Now users can instantiate basic graphics the same way they'd instantiate other widgets. We can also cache graphics data in more clever ways now that graphics elements (now widgets) are a part of the same graph as the rest of the widgets. We also no longer require a draw method for Widgets, as their graphics can be created by instantiating child primitive widgets in the update.

Interesting. My guess is, that because imperative event handling does not compose as nicely as FRP does, that it's impossible to write stateful widgets in the same manner as stateless declarative elmesque primitives.

Anyhow, I'll write up my architecture ideas for declarative UI components (or widgets) soon. This should give us more concrete grounds for further discussion. But the general direction is Carboxyl streams & signals combined with something adapted from the Elm and Cycle.js architectures.

[mitchmindtree]

@aepsil0n hey there, I'm going to close this as the discussion seems to have died down for now.

If you do end up having a play around with declarative FRP style UI, please ping me! Still very much interested to see what approach you take, whether or not conrod turns out to be suitable 👍

[milibopp]

Uhm, yeah, I neither have the time nor a use case right now to drive this forward. So don't let this clutter your issue list for too long. ;)

Will let you know, if I ever pick it up again.