Bevy UI and Scene Evolution Proposal

[cart]

Introduction

My highest priority right now is to prepare Bevy's UI system for the Bevy Editor (and a secondary priority is improving the Bevy Scene system). For the past week I have been deep diving on the prior art in this space, experimenting, and consolidating my thoughts on the direction we should be going in.

I have looked at 3rd party Bevy UI frameworks like kayak_ui (author: @StarArawn), belly (author: @jkb0o), cuicui (author: @nicopap), and bevy_lunex (author: @IDEDARY). I'm combining that with my thoughts and opinions I've developed via professional experience with webdev (largely React and Angular).

I've also looked into bevy_proto (author: @MrGVSV): a surprisingly feature complete bevy scene system.

The TLDR is that I think we should take the best ideas from the frameworks above (especially kayak_ui, bevy_proto, and belly), add in a few new ideas and unify them into a single cohesive whole. I've started building a prototype to prove this out (which I have linked to below). My goals for this discussion are:

1. Share where my head is at after this first week of investigation
2. Outline a rough plan for the future to kick off the discussion (with prototypes).
3. Iterate with the community until we have something we all believe in. I want to get as many experts on board as possible (especially the authors of the libraries above) so we can build toward the future together.

Here are some general thoughts and goals:

• Unified and Bevy-Native: We need a nicer way to define UIs and scenes, both in code and in external assets. The existing Command spawn APIs (and ..default syntax) are a pain point for many people / don't scale/nest well. There is way too much overlap to treat UI and Scenes as different things. Defining UI should be done using the Bevy Scene system. It should build on top of (and feel at home alongside) existing Bevy pillars like Bevy ECS, Bevy Assets, Bevy Scenes, Bevy Reflect, etc.
• A New Bevy Scene Format (BSN): Rust developers should feel "at home" composing it, but it should be as terse as possible. Scenes must be human composable. It should be possible to cleanly express hierarchies of entities, with each entity in the hierarchy having zero to many "things" (I'm saying "things" instead of Components on purpose). Existing formats like XML aren't ideal because we want to cleanly support hierarchical entity nesting and adding multiple "things" to an entity. New syntax can make this much clearer. XML also doesn't map very well to Rust or to Bevy ECS. The current Bevy Scene format (while much better than it used to be) is not human-composable to the level it needs to be, nor would it be a good fit for defining scenes in rust code.
• Code and Asset Driven Workflows: Developers should be able to compose Bevy Scenes in code or in asset files. Asset scenes and code scenes should use the same syntax. Code scenes should be able to use asset scenes (and vice versa).
• The bsn! proc_macro: To accomplish this unification, we need a proc macro that can produce "bevy scene rust code" from the bevy scene format. This macro should also enable passing data from systems into the scene.
• Efficient Code-Driven Scenes: When defining scenes in code, it should be efficient. It should be as close to the cost of spawning Entities/Components/Bundles using direct World apis as possible. Basic scenarios (spawning an entity with a schematic / spawning children) should not involve unnecessary costs like allocations, hashing, or reflection.
• IDE Integration: When defining scenes in code, users should have IDE features like autocomplete, go to definition, and syntax highlighting. Rust Analyzer must play nicely with the bsn! macro.
• Scene Inheritance / Nesting: It must be possible for a given entity in a scene to inherit from a separate scene (either defined in code or in an asset file). This should include children. It should be possible to inherit from more than one scene (more on this below).
• Property Overrides: It should be possible for an entity inheriting another scene to override specific properties in that scene. For example, you should be able to spawn multiple Rock scenes in a parent scene, and override the default position of each rock. Other engines like Godot embrace this "overriding" pattern (although not to the extent that I am proposing).
• Cascading Properties: Properties should "cascade". If a scene defines a Style::color property on an entity, it should override that specific property in any scenes inherited by that entity, but keep the other Style properties defined in the inherited scenes. This isn't something unique to Styles. All scene properties should be override-able / cascade-able. If inheriting from multiple scenes, properties should "cascade" across scenes in the order the scenes are referenced
• Styles are Scenes: I don't think we need a separate style sheet system. This "cascading" scene overriding / inheritance is already a style system. Just inherit from zero to many "style scenes" and override any fields that you want. Multiple scenes will "cascade" in the order they are referenced (much like CSS). However unlike CSS, this is "fully local" (no global style overrides from arbitrary sources). Modern web frameworks have started to head in this direction and I also think it is a good idea. We can still support "theming" scenarios using the component-theming patterns adopted in frameworks like React. I don't think we need a new "style sheet" system. Scenes also have the benefit of being fully extensible. We don't need to limit ourselves to a fixed set of Style properties. Developers can define their own Schematics and use them as styles using the exact same system as the built in Bevy UI system styles. Good luck doing that in CSS! I personally think this unification is an extremely powerful (and maybe novel?) idea. Hyper flexible while also reducing conceptual load.
• Schematics: We need a new scene data model that maps to the runtime data model (very much like Schematics in bevy_proto). Schematics define arbitrary input data that is then fed into the ECS World to produce arbitrary outputs. Schematics, when being spawned, have access to the "current" entity and other scene metadata. This allows them to do things like "convert an asset path to a strong asset handle" or "convert an entity path to an actual Entity value". Schematics will largely fill the role that Bundles currently fill in Bevy. In fact, we will generally just add a Schematic derive to things like PbrBundle and SpriteBundle (although we can and probably should discuss adopting shorter names for schematics).
• Props / Properties: To enable the cascading and overriding features, Schematics fields should be Props, which are values that can be set (or unset) much like an Option type. But they have an apply function that performs the "cascading" behavior.
• Hot Reloading: We need to be able to hot reload asset scene changes (and apply those changes to everything that inherits from them). This should preserve existing entities / components whenever possible. We should also investigate "code hot reloading" in two forms:
  1. Because our code-driven format and asset-driven format are identical, I believe when in "dev mode" the bsn macro can register the source Rust file with the asset system to watch for changes. We can then hot-reload any changes that don't add new rust expressions to the bsn macro. This is inspired by the makepad approach.
  2. We can also look into dynamic reloading of Rust libs, which would allow "fully" reloading of all logic, but this is much more of a research project (but we'll be researching this anyway for the editor).
• Reactivity: We should have optional "reactivity" for scenes, fed using ECS data. This is largely motivated by UI, but I see no reason to limit it to that. This can likely build on some of the features from the hot reloading system (ex: entity stability across reloads, determining what to spawn and what to remove, etc). This should be roughly like the kayak_ui implementation, but there are improvements to be made (such as improving how we reload scenes when inputs change and making prop change detection more automatic and user-friendly).
• Entity Events: We need a Bevy ECS entity event system, with the ability to bubble events in the hierarchy (when necessary). Entities should be able to register their own handlers for events. These handlers should have access to ECS data / be ECS systems themselves. I believe bevy_eventlistener is already essentially what we need, so upstreaming that should probably be our goal. Although I think we will want to adapt the event dispatcher code to enable events triggering other events to be handled within the same frame. We will also want to disable event bubbling by default (as most user-defined events won't need bubbling I think).
• Context-free Scene Declaration: Declaring scenes in code should not require (direct) access to AssetServer, Commands, contexts, etc. They should just be data that can later be applied (using those contexts).

Note that I am currently focused on core Scene API design and dataflow. Once we resolve these things, we can focus on higher level concerns like "how will we support rounded corners", "should we use flexbox or morphorm", "what widgets will we build", "what will our Input widget look like", etc.

The Prototype

I have built a (very incomplete, super experimental, barely functional) implementation of many of these ideas, just to prove them out. Note that everything is open for discussion at this point (broad goals, implementations, syntax, etc). This is the result of a week of work and experimentation. It is not the platform we need to build on.

Note that this work is heavily inspired by bevy_proto and kayak_ui, with a little bit of belly and to a lesser extent the other libraries mentioned above. My goal is to take the best ideas from the ecosystem and mash them together with my own for a "best of all worlds" result. Note that I didn't take any code from any of these libraries, just ideas.

Also note that my prototype builds on Bevy Asset V2. This makes use of "direct asset loading" to enable producing final "combined" scenes from nested scenes. And the improved dependency tracking makes the code-driven BSN implementation much simpler.

A New Bevy Scene / UI Format

Here is what my proposed Bevy Scene (BSN) format looks like in its current form. Note that I have a working prototype implementation for everything I am covering here.

Here is a single entity with a single schematic:

Div { width: 10 height: 20 }

Note that the Div schematic I'm using in these examples is not a final api, just a way to illustrate with familiar terminology. Style would likely have its own field on the Div schematic rather than having fields like Style::width splatted onto Div.

Fields are entirely optional:

Div { width: 10 }

You can even skip everything!

Div

You can nest child entities like this:

Div [
    Div { width: 10 }
]

You can define fields and nest like this:

Div { width: 10 } [
    Div
]

Most entities will only need a single schematic. But you can add more than one using a tuple (just like we do in Bevy Rust code for Components):

(Div { width: 10 } Player { health: 100 })

You can assign names to entities using this syntax (very TBD). It will produce a Name component for the entity:

#Player1(Div { width: 10 } Player { health: 100 })

You can add the contents of a scene to an entity like this:

@"player1.bsn"

This is how you would add a scene as a child of another entity:

Div [
    @"player1.bsn"
]

You can override schematic values in scenes like this:

(Div { height: 200 } @"player1.bsn")

Assuming the player1.bsn scene contains the #Player1 entity defined a few code blocks up, this would result in a final scene that is:

(Div { width: 10 height: 200 } Player { health: 100 })

I was pretty sure that the BSN format does not fit well into Serde data model, so I built a custom parser that produces an AST that gets converted into the final in-memory Scene asset (using TypeRegistry). This means that we currently need FromBsn implementations for every type in the scene. I'm not sure this is the right move yet (I just took the path I knew would work). Serde might still be an option, either in part (ex: use FromBsn when it exists for a type and fall back to serde when it doesn't) or in full (make a serde-compatible BSN deserializer). We could also rely on Reflect for everything but "value deserialization" (at the cost of some overhead).

Using BSN in Rust Code

Scene asset files can be spawned like this:

fn spawn_scene(mut commands: Commands, assets: Res<AssetServer>) {
    commands.spawn(assets.load::<Scene>("player.bsn"));
}

You can also use the bsn! macro to define your scenes inside Rust code:

fn setup() -> Bsn {
    bsn! {
        Div { width: 10 height: 100 } [
            Img { handle: "branding/icon.png" }
            Div {
                width: 100
                height: 200
            }
            @("player.bsn")
        ]
    }
}

Rust Analyzer's autocomplete, import missing symbol, syntax highlighting, and go-to-definition all work because bsn! plays nicely with RA:

You can also pass data in from systems and define expressions inline:

fn setup(score: Res<Score>) -> Bsn {
    let score = score.0;
    let ten = 10;
    bsn! {
        Div  [
            Div { width: ten }
            Label { val: {format!("Score: {score}")} }
        ]
    }
}

I've created a system adapter to spawn the returned BSN (not sure this will be the final api, but its convenient for testing!)

app.add_system(Startup, setup.pipe(spawn_bsn));

Note that the bsn! macro does not generate an Asset file (which would introduce more dynamic-ness and overhead than necessary). It directly creates efficient Prop instances inline, which are then turned into insertable Bundles and Components using the Schematic impl for the type and inserted into a given World. This makes it roughly equivalent to spawning Entities/Bundles/Components directly using World. The goal here is to make it an (optional) replacement for the current command.spawn() pattern people are using today in Bevy.

Defining Schematics (and Props)

The Div schematic used in the examples above is defined like this:

#[derive(Component, Reflect, Schematic, Default)]
#[reflect(Component, Schematic)]
pub struct Div {
    pub width: u32,
    pub height: u32,
}

This generates DivProps:

#[derive(Clone, Default, Reflect)]
pub struct DivProps {
    pub width: Prop<u32>,
    pub height: Prop<u32>,
}

impl Props for DivProps {
    fn apply_props(&mut self, other: &Self) {
        self.width.apply(&other.width);
        self.height.apply(&other.height);
    }
}

Prop is a simple Option-like type that looks like this:

pub enum Prop<T: Props> {
    Unset,
    Value(T),
}

In fact, we might be able to just use Option in the final impl!

Deriving Schematic for Div also produces impl FromBsn for DivProps code, which enables converting from the deserialized BSN AST to DivProps.

If you add the #[schematic] attribute to a field, it will treat the Prop as a schematic instead of "just" a prop value:

#[derive(Component, Reflect, Schematic, Default, Debug)]
#[reflect(Component, Schematic)]
pub struct Img {
    #[schematic]
    handle: Handle<Image>,
}

Handle implements Schematic and uses AssetPath as the Props impl:

impl Props for AssetPath<'static> {
    fn apply_props(&mut self, other: &Self) {
        *self = other.clone()
    }
}

impl<T: Asset> Schematic for Handle<T> {
    type Props = AssetPath<'static>;

    fn from_props(
        asset_path: Self::Props,
        context: &mut SchematicContext,
    ) -> Result<Self, SchematicError> {
        Ok(context.assets.load(asset_path))
    }
}

Note that schematics don't need to be components or bundles. They can "just" be fields.

Reactive Systems (Not Actually Reactive Yet ...)

I have not implemented reactivity yet (and this will be a challenge!). However I have wired up a pattern that I believe could be made reactive:

#[derive(Component, Clone, Schematic, Reflect, Default, Debug)]
#[schematic(example)]
#[reflect(Schematic)]
pub struct Example {
    score_multiplier: Prop<u32>,
}

fn example(In(props): In<Example>, score: Res<Score>) -> Bsn {
    let score = score.0 * props.score_multiplier.get();
    bsn! {
        Div [
            Label { val: {format!("Score: {score}")} }
        ]
    }
}

fn setup() -> Bsn {
    bsn! {
        Div [
            Example { score_multiplier: 2 }
        ]
    }
}

The idea here is that we can use the ECS to change-detect whenever the input props change (because they are stored on a Component, which has automatic change detection). We could also probably add a new IsChanged trait for SystemParam that we implement for change detectable parameters like Res, which would enable us to re-compute the scene when score changes.

Not sure this is the best path forward yet. But I think it is a reasonable start.

Missing Features / Next Steps

Please note that this is just a prototype. In its current form it is not usable for anything. Too many missing features!

Nested Entity Overrides

It should be possible to override values in child entities

// foo.bsn
Div [
   #A(Div { width: 10 height: 10 }) 
]

// bar.bsn
Div [
    @"foo.bsn" [
        @A(Div { width: 20 } )
    ]
]

Proper Hot Reloading of Children

Hot reloading currently works, but it will create duplicate children on each reload. We need to add logic to write on top of entities whenever possible, preserve moved entities (if they have the same name), and ensure we never create duplicates. Non-trivial! I suspect we can use this logic for reactivity as well.

Resolve "relative entity paths" in Schematics

It should be possible for an EntityPath Schematic to exist that allows entities to reference other entities in the hierarchy:

#A(Div Foo { c_entity: "B/C"} ) [
    #B(Div Bar { c_entity: "C"}) [
        #C(Div Bar { b_entity: "../B" })
    ]
]

We also need to decide what restrictions we will place on this. Allowing arbitrary access to the hierarchy means we need to spawn all entities in a scene before we insert components. If we only allow accessing children, we can directly spawn "bottom up". Will we allow referencing entities "outside" of a scene?

For Loops in BSN Macro

We need to be able to generate children from iterators over inputs

bsn! {
    Div { width: 10 height: 10 } [
        Img { handle: "pretty.png" }
        for item in items {
            Div { width: item.width }
        }
    ]
}

Reactivity

As mentioned in the "reactive but not actually" example, this is a big chunk of work that needs to be done. We can learn from kayak_ui here, as well as more complete / battle tested reactive systems such as React, Solid.js, etc.

Versioned Imports and Migrations

Currently "asset bsn files" use the "short name" as defined in the TypeRegistry. This, on its own, is not a safe approach to referencing Schematics. Every schematic must have a version number associated with it so we can properly migrate it across Bevy versions. I think we'll want something like "versioned imports" to encode this. It should not be possible to access a type in a scene file without specifying the version (or at least, we should warn about this on load). For simplicity, we likely want a "bevy scene prelude" that pulls in all built in bevy schematics:

use bevy(0.11)::*

PbrBundle { mesh: "some.gltf#Mesh0" } [
    PbrBundle { mesh: "other.gltf#Mesh0" }     
]

We'll also want some way of resolving conflicts (ex: import aliases and/or fully qualified schematic names).

Deriving Schematic for Enums and Tuple Structs

#[derive(Schematic)]
struct Foo(String);

#[derive(Schematic)]
enum Foo {
    Bar,
    Baz,
}

Improved Errors

The BSN parser does not currently provide line information / "error location context" in the source asset. This is a must have.

Nested Code-Driven Bsn Scene Dependency Tracking

Currently when nesting a code-driven Bsn inside of another code-driven Bsn, I don't properly wait for scene dependencies in the child to load before trying to spawn the parent.

ToBsn

We'll need to be able to serialize Schematics to BSN (my prototype can only read the format).

Other Missing Features

• Enums and tuple structs are largely unimplemented or incomplete, both in the BSN asset parser and in the bsn! macro.
• The Name Syntax is not supported in the bsn! macro yet
• Separate Props derive. Currently the only way to derive Props is via Schematic. But for nested types we'll probably want non-schematic prop derives.

Open Questions

• Reflect + Schematic + Props + ToBsn + FromBsn derives for every Component (and Bundle) is a lot of codegen. We should make sure we are generating the absolute minimum amount of code required for all of the functionality we need. I believe Props remove the need for FromReflect (and therefore also Default) derives on components and bundles.
• Some might look at the Props generated in the Schematic derive (and the Prop-cascading system generally) and think "@cart why didn't you use Reflect for this?". I'm reasonably certain we could do this if we really wanted to (ex: something like Reflect::apply + DynamicStruct with Options wrapping values). I chose not to follow this path because that would incur some pretty serious overhead (hashing, string comparisons, dynamic dispatch, etc), making the system an unsuitable replacement for code-driven scene spawning (and making our asset-driven scene spawning less efficient too). I'm open to increasing our usage of Reflect (ex: maybe for removing the need for FromBsn derives everywhere), but not if it comes at the cost of good performance. I'm reasonably certain that Props are the right tool for this job. Reflect will still be useful for things like entity inspectors, auto-generating widgets for scene/schematic editors, migrations, diffing rust types for dynamic lib reloading etc.
• Layout: should we move to morphorm for layout and drop flexbox? It has a lot going for it.
• Do we like the BSN syntax? What should change?
• Can we use the proc_macro parser for both the bsn! macro and the scene format? Should we? It would ensure both behave exactly the same.
• Belly has the concept of "data bindings", which are built on a generic "ECS data binding system". This is a powerful concept that seems like it could be useful, but I'm also a bit worried about the complexity effects such a system would have on Bevy code (individual entities' components having arbitrary dependency arrows pointing anywhere in the ECS, with arbitrary graph-like dependency chains). We should decide if this is something we want in an MVP UI framework. I think the answer is no (in the context of UI) if we have "reactivity". Reactivity is a bit more principled + hierarchical, but it is also more limited! In a world where we have prop-driven reactivity, I see data binding as being more relevant to non-ui scenarios. My inclination is to table arbitrary data binding for later and focus on prop-driven reactivity for UI.
• We can (and probably should) support inline assets and resources in scene files. This could probably accomplished via schematics (ex: a Res<T: Schematic> schematic and an Asset<T: Schematic> schematic), but we might also want special syntax for it.

Call to Action

I've made this post as early as possible because I think our first priority should be synchronizing vision and consolidating efforts. Please read through this (especially if you are one of the "3rd party bevy plugin developers" listed in this post) develop your opinions on where we should be headed, and respond here.

[TheRawMeatball]

Quite exciting! Overall, I like the ideas that are present here, but would like to highlight some I think are important.

First, I want to emphasize the question around whether we want data bindings: The existing examples show promise as a way of deriving an inactive scene composition from existing game state, and keeping it up to date with said game state, but there's nothing about UI feeding back into the game state - not even a button. I'd like to see an example of how a button works under this framework, where it seems scenes are our "reusable widget abstraction", and ideally some more complex examples featuring where two-way data binding shines, such as a textbox or a slider, if anything so we know what tradeoffs we're making.

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

Third, another concern is how the current examples lack anything which could be considered "UI state". It seems the entire UI is derived directly from game state. For various things like animated widgets, we will most likely require a way for UI to have and manage some state of its own. This state could (and in my opinion should) be stored in the ECS at the end of the day, but its still a conversation that needs to be had, and I'm hoping i can start it.

To wrap things up, most of the points I raised here are stuff I ran into back when I was implementing my last UI prototype https://github.com/TheRawMeatball/ui4, after multiple failed prototypes, and which took a strongly data-binding based approach to things. It is old code, based on an ancient version of bevy, and ugly as sin, but it was feature-complete enough to implement some simple animations and a TodoMVC example, so perhaps it's still worth a look.

[hymm]

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

What I liked for this in web was css variables. For the bsn format I'd imagine something like:

// some "widget.bsn"
Div { --primary_color: red secondary_color: blue } [
    Div { color: var(--primary_color) }
    Div { color: var(--secondary_color) }
]

(primary_color: orange @"widget.bsn")

syntax is a placeholder. not sure what limitations we'd have here

[cart]

but there's nothing about UI feeding back into the game state - not even a button. I'd like to see an example of how a button works under this framework, where it seems scenes are our "reusable widget abstraction", and ideally some more complex examples featuring where two-way data binding shines, such as a textbox or a slider, if anything so we know what tradeoffs we're making.

Yup I agree this is a gap. Its on the "todo list", but I neglected to include it in the "missing features" section.

The general idea is to utilize something like bevy_eventlistener for the "event handling" side of things, which would in turn set state that results in a re-computation of the scene:

#[derive(Component, Clone, Schematic, Reflect, Default, Debug)]
#[schematic(my_button)]
#[reflect(Schematic)]
pub struct MyButton {
    width: Prop<u32>,
}

pub struct Clicks;

fn my_button(In(props): In<MyButton>, clicks: Res<Clicks>) -> Bsn {
    let clicks = clicks.0
    bsn! {
        (
            Div { width = props.width } 
            OnClick(|event: Listener<ClickEvent>, mut clicks: ResMut<Clicks>| clicks.0 += 1)
        ) [
            Label { val: {format!("Clicks: {clicks}")} }
        ]
    }
}

Note that I used a Resource here for simplicity (and how it easily ties into the IsChanged SystemParam trait I discussed), but I think it should be possible to store this state in components on other "state entities" and/or have the ability to store it on the UI entity itself. Worth exploring!

I think the desire for "for-each" style systems increases with reactive UI. It would be nice to be able to express:

fn my_button(In(props): In<MyButton>, clicks: QueryItem<&Clicks>) -> Bsn {
    let clicks = clicks.0
    bsn! {
        (
            Div { width = props.width } 
            OnClick(|event: Listener<ClickEvent>, mut clicks: QueryItem<&mut Clicks>| clicks.0 += 1)
        ) [
            Label { val: {format!("Clicks: {clicks}")} }
        ]
    }
}

Where QueryItem is populated by some my_button.run_on_entity(world, current_entity) (and it could also implement IsChanged).

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

This is a reasonable take. "reactive" scenes are currently "input prop encapsulated", meaning that it is not possible to override anything in the internal schematics (it must go through props).

We could consider adding something similar to asset files, but I suspect in the context of "editor / asset driven scenes" arbitrary nested override behavior will generally be more desirable.

Third, another concern is how the current examples lack anything which could be considered "UI state". It seems the entire UI is derived directly from game state. For various things like animated widgets, we will most likely require a way for UI to have and manage some state of its own. This state could (and in my opinion should) be stored in the ECS at the end of the day, but its still a conversation that needs to be had, and I'm hoping i can start it.

Agreed (see my reply to the first part of your message).

To wrap things up, most of the points I raised here are stuff I ran into back when I was implementing my last UI prototype https://github.com/TheRawMeatball/ui4, after multiple failed prototypes, and which took a strongly data-binding based approach to things. It is old code, based on an ancient version of bevy, and ugly as sin, but it was feature-complete enough to implement some simple animations and a TodoMVC example, so perhaps it's still worth a look.

Oops yeah it totally slipped my mind that you did all of that experimentation. Sorry about that. Reviewing that is now at the top of my stack!

[nicoburns]

+1 for no nested overrides, and making "everything props" such that if you want to override something deeply nested then you need to pass it through explicitly. We could also have something like React's "context" which is available for an entire subtree, but still requires the nested component to explicitly expose the thing you want to change as a parameter.

If that doesn't work for "scenes" then I'd take that as evidence that we shouldn't have a unified API for scenes and UI.

[JMS55]

Second, I have some concerns about scene files sometimes being too transparent an abstraction - e.g. consider someone making a "Fancy Div widget" composed of a bunch of divs. I feel it should be possible to expose certain values as "These are here to be overriden" without requiring upstream users know the exact location of these values in the underlying widget. So, perhaps some form of function-ness to widgets. (Though worth noting, this is the point I'm least confident in, due to a lack of experience)

Generally I think that should be solved by widgets-as-functions / allowing nodes as props themselves. Example (ignore the syntax not using bsn!, it's pseudocode):

fn main_menu() -> Bsn {
    column(
        text("hi"),
        submenu("foo", button("Exit", Style.red)),
        submenu(SubmenuProps { // Different syntax, struct as single arg instead of many arguments
            label: "foo",
            slot_foo: button("Exit", Style.red),
        }),
    )
}

[ickk]

In the scene format, I think we should consider giving props to scenes themselves as top-level values that can be referenced at arbitrary places within the scene. This makes scenes more "function-like" giving them inputs, and potentially simplifies the data flow (you could avoid implicit cascading. Cascading seems tricky because for instance we want to support selectively cascading a pair of different Colors to alternating children).

Scenes with top-level pluggable props also start to look like something that can be modelled and composed using a visual node-graph editor, which may be a usecase we want to support in the future.

Additionally, consider syntax for doing arithmetic in scene definitions, similar to css calc & var. A child may want to inherit some (or multiple) values from its parent but with some adjustment. I think this needs to be expressible in the scene format directly if we expect users to write & style UIs.

[CooCooCaCha]

Regarding calc I wonder if we could make it even more general-purpose by allowing scenes to "request" a value from whatever system is loading the scene. So for calc we could have a MathProvider that you register with bevy before loading a scene. Then scenes can use functions provided by MathProvider inside the scene definition. In this example, any time calc() is called MathProvider parses it and returns a value which is then injected into that spot.

[alice-i-cecile]

This is a feature where we'll want to think carefully about the exact powers here. Allowing full eval powers is likely to lead to both spaghetti and security concerns.

[CooCooCaCha]

I agree, although I think it’s worth discussion. Off the top of my head I can think of quite a few useful helpers like math helpers (min, max, calc), color helpers (darken, lighten, color mixing), localization helpers (string lookup, date/time formatting), custom game specific helpers, etc

Now that I think about it, this is similar to helpers in templating languages like handlebars. But I’m definitely not suggesting we go full templating language.

[alice-i-cecile]

Makes sense. It feels like the right way to do this is to define magic strings in your app that cause a call to a fixed Rust function that can then be inserted into your assets. Infinitely flexible, but still completely in the App's control.

[TesseractCat]

One way to add interactivity could be to use a functional hypermedia approach. Having systems/functions that return scenes, and then having components that connect to them and describe some sort of swapping strategy. This is inspired by the HTMX library.

I imagine this working something like this:

// Could query to get the element to be swapped, the element that triggered the swap, etc.
fn example(trigger: Query<(&SwapTrigger, &Example)>, score: Res<Score>) -> Bsn {
    let score = score.0 * trigger.single().1.score_multiplier.get();
    bsn! {
        Div [
            Label { val: {format!("Score: {score}")} }
        ]
    }
}

fn time(t: Res<Time>) -> Bsn {
    let time = t.elapsed_seconds;
    bsn! { Label { val: {format!("Time: {time}")} }
}

fn setup() -> Bsn {
    bsn! {
        (Div Example { score_multiplier: 2. } SwapOnClick { func: "example" target: SwapTarget::Children }) // Swaps children when clicked
        (Label SwapOnUpdate { func: "time" target: SwapTarget::Self }) // Swap every frame
        (Div { width: 20 } SwapOnHover { func: "foo" target: SwapTarget::Name("/A/B") }) // Swaps neighboring element when hovered
        #A(Div) [ #B(Div) ]
    }
}

This example would require the more advanced entity paths described to be able to reference neighboring/non-children entities dynamically.

[CooCooCaCha]

Honestly this seems way more complex than simply having an if-else-statement that returns different things on hover.

[TesseractCat]

I think the main question is where that if-else-statement should go. There are a few different paradigms being discussed here: Immediate (like bevy-egui), reactive, or hypertext-y (my proposal).

I think this proposal is nice because it integrates pretty well with the ECS, where all UI updates are just systems that return markup. Also no need for a virtual dom or change detection. I'm not sure how well it would generalize though. I imagine more complex UIs would need a reactive approach. But for a lot of standard game UIs (HUD, options, main menu), this might be enough.

[lee-orr]

Just a few quick comments (I'll likely look into things a bit more later, but wanted to jump in with initial thoughts now):

• I like the scene syntax overall, seems fairly readable
• For styles - I think thought needs to be put on inherited styles as well. Cascading is great for importing (the equivalent of having multiple classes on an HTML), but for some things you want to bring things to children (font size & color, for example). Having a "css variable" like behaviour would be useful as well in some cases.
• For widgets/widget-like situations - it might be worth at looking at Web Components for inspiration and the shadow dom - where they expose parts for styling and the like, but generally function as a single node in the main UI hierarchy? This could allow for both UI-only state (like an element being hovered) & explicit styling of sub-parts
• Will there be a UI-world? Essentially making the UI in the main world act as a virtual dom?

[alice-i-cecile]

I'm increasingly sold by the notion of multiple worlds by default, as it makes common operations (reset all UI, serialize all game state for networking or save games) very intuitive. The increased robustness to avoid accidentally grabbing UI / render / window elements in gameplay queries is also nice.

I have no strong feelings about whether they should be truly distinct worlds or just partitions though.

[kaosat-dev]

I agree, currently , the stop-gap solution of placing everything inside specific scenes at the root level can only get you so far , and can get cumbersome ie , adding specific marker components to the root of the UI hierarchy, the root of "level/world" hierarchy, to bulk manipulate them.

[Wcubed]

One concern I have with splitting the ui and "game" world, is that there a re quite a lot of games in which the ui is an integral part of said game world. It's often attached to specific game objects (health bars, command radial menus) or can even be seen by the in-world characters themselves, like holographic screens.
I'm wondering if splitting the ui from the game won't make it harder/more boilerplate-y to do theze kind of things.

And then there are the kind of indu 2d games where the ui is the level. And the character can move across the scrollbars and such. Now for that kind of integration, you could make your own custom ui parts. But to me one of the big selling points of an ecs is that enities aren't inherently "ui" or "2d collidable". Ideally it should be possible to add a set of components to a "button" and suddenly it has a tower defense cannon, while still being a button as well.

[SanderMertens]

Prior art for an ECS based scene format/DSL that supports hierarchies/reactivity/property overriding/variables:
https://www.flecs.dev/flecs/md_docs_FlecsScriptTutorial.html

[cart]

Ooh yeah I definitely should have given this a proper review as part of this effort. Thanks for the link, definitely something we can learn from!

[cart]

For the flecs explorer hot-reloading, are you re-creating entities each time (with new ids) or do you have a method to retain existing entities? If so, what algorithm are you using? I'd like to hot-reload scenes in a running app without removing existing entities (and ideally manually created components) whenever possible.

[SanderMertens]

The gist of it is:

• The deserializer keeps track of all entity ids it creates
• Named entities are first looked up by name, and created if they don't exist. This is the default and is also used for things like hot plugin/component/system reloading
• For anonymous entities, check if the id was already issued by the deserializer. If not, (re)use it. If the id was already issued, create a new id and keep track of a mapping from serialized id => actual id
• Any relationships/components with entity fields that store anonymous entities use the mapping for id translation

The nice thing about this approach is that it works for both clean & populated worlds. There are no guarantees that the ids of anonymous entities remain stable, but it does guarantee that the deserialized result is otherwise equivalent.

[cart]

Much appreciated! What would happen if there are two anonymous entities that are "completely different" and the user just "reorders" them in the hierarchy? (where hierarchy order matters, such as a UI hierarchy)

[SanderMertens]

I'd store an incrementing counter in a separate component (that's also serialized), and use a sorted query to access the UI widgets. Using entity ids for ordering is hard to get right (I think), lots of edge cases even outside of hot reloading, like dynamic UI widgets that receive recycled ids.

[vultix]

I feel like it would be useful to have a list of high-level goals for Bevy UI, as it can help frame many of these discussions. An example of a goal I'm unsure of is how general purpose we want Bevy UI to be - should it be focused on video game UIs, or should it be a contender for the de facto Rust GUI framework?

I'd also love to know the relative importance of these various goals. For example, to what degree do we prioritize ergonomics over performance, or vice versa?

To help facilitate this, I've gathered a list of goals and non-goals from various other UI frameworks. These aren't in any particular order, and I'm sure I'm missing many, but I feel it's a good start.

• Familiar - using concepts from other UI frameworks, potentially building on existing tools (HTML, css)
• Simple - easy to use, small learning curve, etc.
• Concise - how many lines of code to create a todo app
• Flexible - if you dream a UI, can you create it in Bevy UI?
• Responsive - fluidly respond to changes in aspect ratio, resolution, scaling, e
• Internationalization - Easy to use, robust I18n support
• Accessibility - Should be the default, not extra work to add
• Gaming UI framework - well suited for building game UIs
• Generic UI framework - well suited for non-game UIs. Could we build an IDE with Bevy UI?
• Hot reloading - how quickly can you make changes and see them reflected in the UI
• Prototyping speed - how quick is it to throw together a quick UI prototype
• Portable - web, mobile, desktop, console support, potentially embedded as well
• Styles / Theming - ease of making pretty UIs, optionally supporting multiple themes (light / dark)
• Performant - 30fps, 60fps, 120fps, more?
• Extensible - if something isn't supported out of the box, how easy is it to add
• Modular - the ability to use some parts of Bevy UI and combine them in new ways
• Native UI - potentially mimic native look & feel?
• Power efficient - low battery usage, low CPU usage, etc
• Error resilient - how easy is it to panic? How do you recover from errors?
• Small binary size
• Low level vs High level - will Bevy UI only be the building blocks or also have high level components?
• Easy painting API - support a html canvas-like api for easy 2d painting
• Beautiful - it'd be great if the "out of the box" look and feel was beautiful (definitely very subjective)
• Animations - does it easily support performant animations?

It may also be worthwhile doing a survey of which frameworks make which tradeoffs. egui nails the simple and concise goals, at the tradeoff of efficiency and flexibility (due to using immediate mode). Is there any possible way to retain that simplicity? I've personally found it incredible how much progress I can make using egui.

[alice-i-cecile]

Another goal:

• plays nice with inspector-style tooling

Some non-goals (IMO):

• immediate mode
• web-like APIs

[griffenliu]

Build a bevy ide using the bevy engine like godot😀

[delDelta]

I think this is an accurate list of goals. These also happen to be good goals for the larger goal of a bevy editor and IDE, so I'm glad it's on your list 😄

[aristaeus]

Can we use the proc_macro parser for both the bsn! macro and the scene format? Should we? It would ensure both behave exactly the same.

You can solve this by putting the parser in a separate crate (e.g. bevy_bsn_parser). Then bevy_bsn_macro and bevy_bsn can both depend on it. If you avoid circular dependencies you could put the parser in bevy_bsn and import that, but it might include other stuff and increase your proc macro compile time ;)

[B-Reif]

Syntax

Do we like the BSN syntax? What should change?

I appreciate the terse expressiveness of these examples. It might be a little too terse; for example, I'd like a separate syntax for 'override' and 'child'. Maybe something like the familiar 'spread' operator:

fn setup() -> Bsn {
    // easier to spot 'override' at a glance
    bsn! {
        Div [
            Div {
                width: 100
                ..@("player.bsn")
            }
            @("player.bsn")
        ]
    }
}

In the "Nested Entity Overrides" example, this code looks like it spawns children under foo.bsn.

// bar.bsn
Div [
    // What is this?
    // A div, whose children I am overriding?
    // Or another node, which I am providing children to?
    @"foo.bsn" [ 
        @A(Div { width: 20 } )
    ]
]

In React, some 'override-like' configuration is often achieved by using props which accept React elements, just like children but under a different key. This allows a React component's API to define what exactly is or isn't configurable. In our case we might try something like this:

// `title` and `content` are props which take `Bsn`, the same type as `children`
bsn! {
  Header { 
    title: Span { width: 100 } [ "My Title" ]
    content: Div { width: 200 }
  }
}

This also gives a natural way of 'referring' to other entities/elements, by just exposing them as part of the API.

Generally I think what is or is not "overrideable" should be a first-class part of how our data is defined, and the cleanest way to do this would be through the type system.

Terminology

At risk of premature bikeshedding, I think some early terminology might make this stuff easier to talk about. Specifically I think we should have terms for:

• The syntax for declaring UI.
  • In this post, "BSN", which is well-defined already.
  • In React, this is analogous to "JSX", which is syntax sugar over React.createElement.
• The data type returned by this syntax.
  • In this post, also called Bsn. I'd like to disambiguate it.
  • In React this is an "element".
  • This post seems to describe this data as "scenes". We can bikeshed the exact name but I think it's worth disambiguating the return type from the syntax, e.g. fn my_system() -> Scene instead of returning a type called Bsn.
• Functions which return this type.
  • In React this is a "component" (obviously a name collision for us in an ECS engine).
  • Elm calls this a "view".
  • For us, this is a special kind of system function (like "Conditions").
• The input type of such functions.
  • In React, "props". In Elm, the "model".
  • Traditionally React has distinguished between "props", which are basically immutable from the POV of a given React component, and "state", which is subject to change.
  • In our case, we have both the usual SystemParam types of system arguments, as well as the configurable 'props' specified in bsn!. Might be worth exploring this syntax a bit more.

Philosophy

To me, what makes React successful is that it basically models UI as a pure-ish function of app state : UI. If our approach is basically reactive around a return type, that should keep us on the right track. Ideally the 'scene functions' which return Bsn would be akin to ReadOnlySystems, like Conditions are.

This also means we should have a robust model around side effects and impure operations. Something like React hooks would be worth investigating if we're committed to a 'pure-ish' approach.

[Nilirad]

One caveat about the BSN syntax I found, is that a Name can by any String, including for example whitespace and the ( character, which is used to open a tuple of schematics. Surrounding the name in double quotes will make it easier to parse.

#"My (favorite) entity"(Schematic 1 { prop: 10.0} Schematic2)

[iiYese]

Something that worries me that I don't see addressed here is that this looks like you're going to have to pass some context around like egui and the mentioned existing solutions. This has never been plesant to work with especially in ECS because you have this singular construct where everything is coupled fighting with the parallel & adhoc nature of ECS.

If I have multiple separate panels that deal with completely independent things (like for instance in an editor) shoving everything through the same context gets difficult to work with quickly.

[Nilirad]

I wonder if all Bundles can also implement Schematic, or even merge the concepts of bundles and schematics altogether by integrating all the functionality in Bundle.

I recognize that in the current concept, derive(Schematic) may come with a significant amount of generated code. However, I also think that it would be quite convenient, especially for prototyping, to have direct access to bundles and components in scenes without deriving an even one more trait. It also makes the whole system easier to understand, since the user won't have to learn a new concept.

If the amount of codegen is really a concern, I thought about two things that could be done:

• Put the schematic functionality behind a feature flag, on by default.
• Adding a derive macro attribute (I don't remember the exact name), to surgically remove schematic functionality from specific bundles. I don't know though if it can be done with components, since they impl Bundle indirectly.

[cart]

I think we might want the reverse actually (derive Bundle on Schematics by default).

Bundle is a very specific and very scoped ECS concept. Tacking on a bunch of other "implicit context" onto it would probably be too much.

I think Schematic (or whatever we end up calling it) is much better equipped to be the "everything derive". For schematics that don't need to implement Bundle, we could have an opt-out.

[Seldom-SE]

This is interesting! I'd like to share some of my own experience with these subjects, since there's a lot of overlap. At work, I've co-created an asset format with similarities to BSN. It's called bauble, and it's open source, but it's mostly undocumented. I've also co-created a Rust frontend for Unreal's Slate UI, called mgui, which I've gotten permission to discuss. It uses bauble for markup, and it has its own system for reactivity. For context, here's an example/explanation:

// Bauble files are checked for correctness, parsed, and serialized to binary by our asset system at
// build time. These are type imports from `mgui`. You can import any type that implements
// `FromBauble`, or any asset.
use mgui_backend::{
    script::ExplicitScript,
    ui::Ui,
    widget::{Border, Button, Column, Row, Text}
};

ui = Ui {
    // The $ references an asset from the scope. In this case, it's a Rust script (which may instead
    // be written inline).
    init: $init,
    // Attributes are usually used for optional parameters
    content: #[padding = 5, width = 2] Border(Row([
        // In bauble specifically, you can replace certain attributes and field with scripts. The
        // `ExplicitScript($color)` is a workaround. Normally, it'd just be `$color`.
        #[color = ExplicitScript($color)] Button {
            content: #[color = #FFFFFF] Text("Toggle Me"),
            press: $select,
        },
    ])),
}

// `copy` means the value is copied wherever referenced. Without `copy`, the full path of the
// referenced asset is given to `FromBauble` instead.
//
// The `#{}` syntax is for raw values, which are basically strings. The multiple `{}`s are to escape
// the inner `{}`s, similar to Rust's `r##""##` syntax.
//
// These Rust scripts are compiled to wasm at build time
copy init = #{{
    struct State {
        selected: u8,
    }

    ctx.insert(State {
        selected: 0,
    });
}}

// This is a generator, so it returns a value. Generators may generate bauble markup for UI. We use
// a `ui!` macro that's basically the `format!` macro, but it may have inline expressions.
copy color = #{{
    match ctx.local::<State>().selected {
        true => "888888",
        false => "CCCCCC",
    }
    .to_string()
}}

// This is an event, so it accepts a value, which is `()` in this case. It knows this because it's
// used in a button.
copy toggle = #{
    let selected = &mut ctx.local_mut::<State>().selected;
    *selected = !*selected;
    ctx.temp_hardcoded_function_to_react_to_the_selection_reactivity_is_hard_lol(*selected);
}

Here's Button's definition. It's split up weirdly because of how scripts are processed, but types don't have to use weird generics to implement FromBauble.

#[derive(Clone, Debug, Deserialize, FromBauble, Serialize)]
#[bauble(
    bounds = (
        C: for<'a> FromBauble<'a>,
        Co: for<'a> FromBauble<'a> + ColorTrait,
        P: for<'a> FromBauble<'a>
    ),
    path = mgui_backend::widget,
)]
pub struct Button<C, Co, P> {
    /// Child widget
    pub content: C,
    // Alignment of the child widget
    #[bauble(attribute, default)]
    pub h_align: Alignment,
    #[bauble(attribute, default)]
    pub v_align: Alignment,
    #[bauble(attribute, default = Co::WHITE)]
    pub color: Co,
    /// Event to fire when the button is pressed
    pub press: P,
}

// And in some other module somewhere

// `Generator`'s first generic is the type specified in bauble. The second is the type that a script
// should return.
pub type Button =
    mgui::widget::generic::Button<Box<MarkupWidget>, Generator<Color, String>, Event<()>>;

Our requirements are very different, but I hope it's useful to see another similar sort of solution. Anyway, about BSN and bevy_ui:

BSN

I like that you avoided XML syntax. We used attributes in bauble to avoid having to mix properties with child widgets, and it looks like you're going with something similar. I think I like that you have them appended to the name instead of prepended.

It's great to use the same syntax in the asset files as in code. Unfortunately, though, asset files tend to be more limited. In BSN's current design, it doesn't seem like you could, for example, make buttons with Rust closures, so I think people will prefer to use code when possible. We have the opposite problem, though, where buttons must use a wasm script defined in the asset system. If you could use wasm for the asset system and ECS stuff for runtime, you might be able to get best of both worlds. But wasi (wasm but not on the web) is such a mess to work with, so it might not work for you. UI without some sort of scripting tends to be very constraining, so I hope you find a good solution.

We have type checking on our bauble assets, and it's very nice. I'm not sure how you could do that in Bevy/BSN, though, since assets are loaded at runtime after startup.

I enjoy FromBauble, so I think FromBsn can be nice. We have a bauble(flatten) Rust attribute for structs with a single non-attribute field and enums whose variants each have a single non-attribute field. It lets you write the bauble for such fields directly, without a wrapper of a struct or variant, but you can still parse the attributes for the wrapper. It gets used a lot. It's how anything that accepts a color can instead accept a script that outputs a hex code for a color.

#[derive(Clone, Debug, FromBauble)]
#[bauble(flatten)]
pub enum Generator<L, S>
where
    L: Debug + for<'a> FromBauble<'a>,
{
    Literal(L),
    Script {
        src: ScriptSrc,
        // This is like `default`, but you may not specify it in bauble
        #[bauble(as_default)]
        _s: PhantomData<S>,
        // We don't actually have this field on this type, but I've added it to show how attributes
        // work with flatten.
        #[bauble(attribute)]
        my_attribute: String,
    },
    /// If `L` is a `Raw`, it will take precedence over `Script`, so use this if you need a script
    ExplicitScript(ExplicitScript),
}

Finally, I like the name "BSN", because I'm reading it as "bosun" in my head, which is kind of cute.

bevy_ui

Reactivity is really hard! In our first attempt at that, we gradually added more syntax to the markup, such as inputs, commands, if conditions, loops, etc, until it eventually became a programming language of its own. There's just always more you want markup to be able to do. A proper scripting language (wasm in our case) helps with that.

When I make UI in Slate, I tend to make generic widgets that accept other widgets and properties as
arguments. In bevy_ui/BSN, a way of specifically exporting certain properties to be filled in externally would be nice.

Bottom text

I like the ideas here. I have a feeling there will be footguns with BSN, just because of how expressive it's planned to be, but I don't have any particular footguns in mind.

[IDEDARY]

Statement

The reason why I created Bevy-Lunex is because I was overwhelmed by all the web-tech-inspired frameworks everywhere.
I'm a Rust beginner. I started learning Rust about a year ago. I finally comprehended how Bevy works just two months ago.
Bevy-Lunex was my personal learning project in which I slowly learned different kinds of concepts of how to handle UI from scratch.
I am saying this because I want to let you know I am not some kind of senior dev veteran, so you can see from which background my feedback is coming and value it as much as you see fit. I won't pretend I am some kind of expert :)

Beginner friendly

One of my main concerns is that Bevy UI should not become overwhelming to the beginner user. Let's paint a scenario in which you are a beginner indie dev who picked up Bevy recently and just finished his game jam prototype. He wants to quickly bash together a main menu and call it a day. The current proposal uses semi-advanced concepts that are not immediately understandable to a beginner. Just by looking at the code, I don't understand how data/logic relation is modelled. I will have to read a lot about it, which is a little discouraging (as the person). Personally, I think that if Bevy wants to become mainstream, it should be attractive to game jammers.

Office suite

I also like the idea that we are preparing the UI to be used for more classic stuff than directly tailored for games. Bevy already has a low power consumption mode. That can make it a very good choice for regular desktop applications, not just games. I am very excited about this.

Data relation

I am also curious about how we can manage data across the divs. Rusts borrow checker is a little stubborn when it comes to UI and we need to dance around it. It doesn't let us store pointers to other data types (obviously), so I wonder what other pattern will we use access data A in div Foo from div Bar. Like buttons for example. When we click on a div button we expect it to trigger something that will change other divs. How do we define which div precisely should change? I solved this by using a tree-like structure hierarchy and by using paths. I'm curious about other takes on this problem.

Logic

Also, just to voice a concern, we have an amazing ECS ecosystem, but web frameworks are not ECS based, so I hope we won't gravitate too much to "non-ECS" concepts. I think I stroke a nice balance with Bevy-Lunex in this regard, where you can use Bevy Systems to query for button entities and add any logic desired. An example can be to enlarge the button image by 1.2 if the cursor hovers over it. For any button that I want to do, I just add the component to it with no changes required. I would like to see this get adopted here in some form.

Closing

I will leave the more technical stuff to be debated by more qualified people than me. I don't have that much experience in that regard. I just rode with everything that seemed to work :D
I am very excited that we are finally getting to start on a proper UI solution. Keep up the good work :D

[iiYese]

Something I want to be a part of this discussion is relation based UI. I'm still working on some changes for the next release of aery that enables this but this is just some dogfood code. The main thing to notice here is that it's an extension of how current bevy_ui works. Where current ECS can only take advantage of row polymorphism with components this takes advantage of sibling polymorphism by analyzing the hierarchy structure.

// This would be part of a UI lib. 
// Can be made completely generic and a write once use everywhere thing with individual widget logic delegated to systems callbacks with one shot systems.
// Otherwise currently requires a marker component + system pairing. 

#[derive(Component)]
pub struct ScrollSpan {
    pub lower_bound: f32,
    pub upper_bound: f32,
    pub view: f32,
    pub pos: f32,
}

#[derive(Component)]
pub struct VerticalScroll;

#[rustfmt::skip]
impl VerticalScroll {
    pub fn update(
        focus: Res<Focus>,
        mut wheel_evr: EventReader<MouseWheel>,
        mut spans: Query<&mut ScrollSpan, With<Self>>,
        widgets: Query<((), Relations<Widget>)>,
        scroll_areas: Query<(), With<ScrollArea>>,
    ) {
        let Some(scroll) = wheel_evr.iter().last() else { return };

        widgets
            .ops()
            .join::<Widget>(&mut spans)
            .join::<Widget>(&scroll_areas)
            .traverse_targets::<Widget>([focus.entity])
            .for_each(|_, _, (mut span, _)| {
                span.pos = (-scroll.y * (span.view / 4.).max(1.))
                    .pipe(|shift| shift + span.pos)
                    .clamp(span.lower_bound, span.upper_bound);
                ControlFlow::Exit
            });
    }

    pub fn draw(
        spans: Query<(&Rect, &ScrollSpan, &Paint, &MainColor, Option<&AltColor>), With<Self>>,
        mut painter: ShapePainter,
    ) {
        for (rect, span, paint, main, alt) in spans.iter() {
            let pill = (span.upper_bound - span.lower_bound)
                .pipe(|denom| (span.view / denom, span.pos / denom, rect.sizes()))
                .pipe(|(t_view, t_pos, sizes)| ((sizes.y - sizes.y * t_view), t_pos))
                .pipe(|(view, t_pos)| (*rect).tap_mut(|rect| {
                    rect.north_east.y -= view * t_pos;
                    rect.south_west.y += view * (1. - t_pos);
                    rect.z += 0.5;
                }));

            for (rect, color) in alt
                .map(|color| (pill, **color))
                .into_iter()
                .chain(std::iter::once((*rect, **main)))
            {
                painter.color = color;
                painter.corner_radii = Vec4::splat(paint.roundness);
                painter.set_translation(rect.center().extend(rect.z));
                painter.rect(rect.sizes());
                painter.reset();
            }
        }
    }
}

// The spawn API can be turned into a UI DSL with some cleanup work
// This is tile based so rect layout is handled by a tiling algorithm but other styling is done via regular components.

commands
    .spawn((Tiling::H, Rect::default(), MainColor(theme.mantle)))
    .scope::<Widget>(|mut rect| rect // Add children of type `Widget` in this scope
        .add(ScrollArea)
        .add((
            VerticalScroll,
            ScrollSpan { upper_bound: 256., view: 16., ..default() },
            MainColor(theme.surface0),
            AltColor(theme.overlay0),
            Paint::default(),
            Rect::default(), 
        ))
        .add((
            HorizontalSCroll,
            ScrollSpan { upper_bound: 512., view: 32., ..default() }
        ))
        .add(ScrollSpan { upper_bound: 512., view: 32., ..default() })
    });

I've modeled a scroll area with this that can represent arbitrary tensor data. You can scroll in as many dimensions as you like by adding more and more children. For my use case I currently only need input for vertical and horizontal scrolling, scroll bar drawing for vertical scrolling & 1-2 more dimensions for some complex widgets.

This becomes even more powerful with fragmenting relations cause they're can be more efficient for queries involving entities that are not immediately local. For UI specifically there's also many benefits to using relations like:

• Being able to create reactivity via queries and change detection (essentially diffing for free).
• Hot reloading assets fitting in much more nicely via assets as entities.

[killercup]

One thing missing from the discussion so far is a mention of the new-ish declarative UI frameworks for mobile, Kotlin Compose and SwiftUI. I'm not sure they match the goals set out above as they are for a different use case, but a lot of thought has gone into their design and implementation. Both use UI code written in the native language, come with data binding primitives, have IDE support with previews, and have inline styling.

Sadly I've also not used either much, so I can't go into detail of what we could learn from them. I'd appreciate it if someone could jump in here and fill in the gaps :)

[nicoburns]

There are also several such frameworks in Rust: Iced, Dioxus, Leptos, etc. IMO you'd want to adapt such approaches to integrate the reactivity with Bevy's ECS, but the core approach of such frameworks seems solid and well thought out to me.

[cart]

I've looked into both of Jetpack Compose and SwiftUI in the past (read through their tutorials), but I don't have any practical experience with them. I'll give them another look!

The Rust frameworks @nicoburns listed are also on my radar.

[databasedav]

there's also Dominator, MoonZoon, and Silkenweb which are all powered by the incredible FRP signals of https://github.com/Pauan/rust-signals

i just started using Bevy and was surprised to learn that the UI is going in the DSL direction when the reactivity + ergonomics of signal semantics can be accomplished without one (even without macros), see the example from the MoonZoon README:

use zoon::*;

fn main() {
    start_app("app", root);
}

#[static_ref]
fn counter() -> &'static Mutable<i32> {
    Mutable::new(0)
}

fn root() -> impl Element {
    Row::new()
        .s(Align::center())
        .s(Gap::new().x(15))
        .item(counter_button("-", -1))
        .item_signal(counter().signal())
        .item(counter_button("+", 1))
}

fn counter_button(label: &str, step: i32) -> impl Element {
    let (hovered, hovered_signal) = Mutable::new_and_signal(false);
    Button::new()
        .s(Width::exact(45))
        .s(RoundedCorners::all_max())
        .s(Background::new()
            .color_signal(hovered_signal.map_bool(|| hsluv!(300, 75, 85), || hsluv!(300, 75, 75))))
        .on_hovered_change(move |is_hovered| hovered.set(is_hovered))
        .label(label)
        .on_press(move || *counter().lock_mut() += step)
}

there has been prior discussion about integrating FRP signals into Bevy here Pauan/rust-signals#31

note i'm completely new to commenting on this level of design, so maybe i'm just inflating the weight of maintaining a DSL or maybe there's some broader game dev benefits of a scene spec that can be defined statically (etc.), but just wanted to mention the rust frontend frameworks that utilize FRP signals given the others listed

[therealbnut]

I came here to mention SwiftUI too.

The library rui might be worth looking at for this, it's not bevy, but it demonstrates some basic SwiftUI-like concepts in Rust.

Having used SwiftUI when it came out, and its predecessors, I'd say it is a notable improvement in its platform's UI ecostystem for:

• Replacing previous mutable representations of UIs with a builder-like DSL that produces a data representation of a UI. Conditional code works much the same way as Bevy's run conditions. Builders can be composed with custom builders to produce new convenient ways to make UI. This data-driven approach allows for functional purity, more efficient state management, and easier reasoning.
• It typically aims to have sensible defaults, so you can focus on expressing only what you care about. This is also platform specific, so because you have different interactions on mobile the same builder might default to slightly different UI. A game example of this would be to have different UI depending on whether you have a controller or keyboard, XBox or Playstation.
• It's driven on-demand by bindings. A builder function produces the UI data representation, the function is re-evaluated whenever the data bindings have changes. Because the UI is represented as data it's much easier to evaluate what has changed and you can relatively efficiently update the internal representation. MobX does cool things here too, but I don't think you could do it with idiomatic Rust. Data bindings work similary to bevy's ECS systems, where a binding is sort of like a Query with Changed.

I think all of this has many similarities to what @cart has described in the introduction, so I believe that it's worth a look.

[lee-orr]

Some additional thoughts:

• is an asset file necessary? Bevy prides itself on being "turtles all the way down", and (at least in the web-dev space) the people implementing UI's are developers themselves nowadays. If got library hot-reload existed, would that suffice?

• Could we use a macro to provide a writing experience similar to immediate mode GUIs, and then separate that out into the "Creation" and "Update" portions? Possibly using a leptos-style fine grained approach under the hood?

[cart]

is an asset file necessary?

The asset file is most necessary in the context of visual editor tooling, especially for large game scenes (outside of a UI context). However I've also really enjoyed composing static UIs in Godot's visual scene editor, so I suspect there will be some overlap here. In the UI space I do suspect that most people will prefer the code driven workflows, but I see no reason not to support both (given that we know we want these "asset scenes" for other reasons).

Could we use a macro to provide a writing experience similar to immediate mode GUIs, and then separate that out into the "Creation" and "Update" portions? Possibly using a leptos-style fine grained approach under the hood?

We could, although going the "immediate mode api + retained state internals" approach feels like a worst of both worlds situation (running the "render" code every frame while also diffing all outputs to check for changes).

[lee-orr]

To clarify - I don't think it'd be a matter of running the render code every frame, but rather of providing a single macro that generates two functions: a "setup" function and an "update" function. The update function would utilize changes to the state (i.e. whatever components & resources get queried) to update the display.

[lee-orr]

(I remember a video by Greg Johnston - the creator of Leptos - that is my inspiration for this idea, but don't remember which. This one might be it: https://www.youtube.com/watch?v=GWB3vTWeLd4&t=1931s)

[cart]

I'll look into leptos!

[TmLev]

While I'm no Bevy expert, I'd like to offer a thought for consideration. While fully integrating Leptos might be challenging, I believe incorporating some of its concepts or components for creating and managing UI could be beneficial.

[viridia]

Question about Eager Evaluation and Styles

I've been puttering around with UI / asset stuff, and I'm trying to decide how eagerly styles should be evaluated.

In HTML, there are two ways to specify a style, either through a class (which implicitly references a stylesheet) or inline. Bevy's current UI system only supports "inline" styles, but I think that we're going to want to have a mix. Both approaches have value in different situations, "inline" styles are great when you want to have a property which applies to a single widget alone, whereas "referenced" or "shared" styles are easier and more efficient when you want multiple widgets to look the same. Under the hood, of course, all styles are "inline", because the rendering engine needs to know exactly what the visible properties are - this is true in browsers as well as Bevy. This simply means that shared styles are copied and merged for each UI element.

In JSX-based systems like React, the "style" is a dynamic expression which can be recomputed every rendering cycle. However, in a world where styles are "assets", this recomputation may be expensive. (Browsers avoid this by doing lots of smart cacheing.)

Assume for a moment that a UI element in Bevy has an array of "style objects" which can either be inline (object literals) or references to assets. Assets are of course specified by their path - the question is, how dynamic are those paths? Can you have a different path every render? Given the asynchronous loading nature of Bevy assets, and the need to check assets for load status, constantly changing the asset path every render is likely to create a lot of churn.

The easy answer is to say that paths are static - hard-coded into the template that calls them, so that once the template is loaded by the asset loader, the asset paths never change. But this eliminates a lot of interesting use cases. It's common in the React world for widgets to accept a class as a parameter, to allow for the caller to customize the appearance of the widget to some degree.

One choice is to declare that style objects can be parameters, but asset paths cannot. That is, once an asset path has been loaded, the resulting asset can be passed around as a parameter to a widget. Asset paths, in this scenario, are eagerly resolved by the asset loader (using LoadContext), which means that the string gets converted into a Handle early on. Once this happens, passing around a string does no good - the assets have already been loaded, the ship has sailed and you missed it. This may seem like a strange limitation, but it is how most labeled assets work.

However, again there are use cases which violate these restrictions. A key one is asset paths that are expressions. Since we're not doing CSS stylesheet-like rule matching, it's helpful to be able to compute an asset path. An example of where this is useful is a widget that has multiple parts, each part with its own style. To be able to parameterize the appearance of such a widget, you could pass in a "base path" as a parameter, and the widget can then join that base path with relative paths that specify the appearance of each widget part.

(A different approach is to introduce some special kind of "theme" object which has a fixed set of slots, which is loaded as a single resource, and which is then passed around as a parameter to the widgets that need it. But let's put that aside for the moment.)

Once you introduce the idea that style asset paths are expressions, then you have to ask the question is to what extent they are different from other expressions. We've all seen the classic "counter" demo, where every time you click on the button the counter increments by one. This model is one where the source of truth for rendering is assumed to change frequently, or at least, arbitrarily. In other words, we have to design our system on the assumption that expressions can change frequently - and that includes references to style assets.

So what's the right balance?

[nicoburns]

Lazily loading style sheets on a per-frame basis doesn’t sound right to me. I’d expect them to be loaded once for a whole “scene”. I suspect something like theme structs ma be the way to go for widgets requiring multiple styles. A path prefix that corresponds to a specific directory structure sounds quite implicit and opinionated to me.

[daveyd777]

This post might not contribute much to the direction, but I just wanted to chime in with some interesting resources for this work.

Ryan Carniato (creator of solidJS) has many in depth breakdowns on their reactivity systems and algorithms. You can get a full breakdown of the DAG algorithm in his video with the creator of reactively (who also has a blog post explaining the full algorithm) . This work largely inspired the reactivity inside of Leptos (rust based solidjs basically), and the creator greg Johnston also has some videos showing a rough implementation of reactivity from scratch on his youtube channel. They use SlotMap underneath to track their reactivity graph. I myself been using these frameworks more, and find them much simpler to reason about than ReactJS. Also far more performant. Although I am not sure how great they would work in Bevy, I am excited about these UI updates and frameworks.

When you aren't bound by the browser and JavaScript, a lot more implementations open up. For me, reactivity is a promising step forward for UI on the web. But I don't have enough experience to speak on the pros and cons of this in a new system from the ground up. Just wanted to share the best resources I have found for anyone interested :)

[dannymcgee]

BSN Syntax Bikeshedding

Sorry I'm so late to the party here, but this is a really exciting development. I think a unified declarative syntax for scenes and UI makes a lot of sense. While I really love the sleekness and legibility of the syntax you've come up with for BSN, I think it's missing one critical thing: labelled closing markers for entities with children.

The need for something like this is really easy to overlook in the prototyping phases, because you're generally putting together very small examples to get a feel for things and prove out the ideas. But I really think this makes a huge difference to readability and maintainability at scale.

The issue arises when you're looking at a large tree declaration with relatively complex nesting patterns. Without labelled closing markers, you end up with a lot of lines consisting of only } or ] where it's not immediately clear which branch of the tree is being closed. In very small examples, it's obvious from the indentation level, because you can just glance up a few lines and read the opening identifier. But that's not always the case in real-life scenarios. When you need to edit an existing tree to insert a new entry at a specific point in the hierarchy, this issue makes it unnecessarily difficult to identify the correct insertion point.

I think probably the strongest argument for this need is to look at battle-tested infinite-nesting syntaxes that don't have this feature. Inevitably you see the same pattern crop up over and over again in the wild: developers insert comments to manually label the closing braces. In the case of Flutter, the official dev tooling even does this for you automatically. I'd really love to see a version of BSN that addresses this from the outset so that this type of hack doesn't become necessary later on.

EDIT: This was a little more controversial than I'd anticipated, which is totally fair. Everyone has their own preferred way of doing things. So, a follow-up thought/question/suggestion: I wonder how feasible it would be to expose some kind of high-level procedural API for constructing a scene description? I.e., in the same way that JSX syntax is just a thin abstraction for React.createElement, if BSN mapped more-or-less directly to a set of publicly accessible Rust functions, it would be pretty straightforward for third parties to build their own abstractions and tooling around it to suit their own use cases. (Then again, if the BSN abstraction is zero-cost, I guess that might not even be necessary?)

[Type1J]

I don't know. I see the QML/Slint like syntax as easier to read, and more like Rust itself than an XML/HTML/XAML like syntax.

How about just making the formatter (I'm assuming BSN has a bsn fmt tool.) have an option to add inline comments after closing braces?

[AlexAegis]

I disagree with using xml, you say you want to see what a closing token closes, and yes thats a given with xml, but as you mentioned you can get the same thing with Json/anything else too, with an extension. And you need/have an LSP anyways so one more little feature is not much. And XML style syntax would need an LSP too, and those need extra editor support too to edit closing and opening tags together.

A simple } closing token like in Json has its own benefits too, its easier to move/swap nodes, as I dont have to worry about moving there closing tokens.

[afonsolage]

I agree, but being optional, not required, so when creating small or medium sized BSN files, one can skip those.

But also, if BSN support comments (which I don't see why not), then this issue can be fixed with a styling guide or some tooling (bevy clippy when?)

[tsal]

please lord, don't ever implement another UI in xml again. the 90s were 30 years ago.

[IronGremlin]

The issue arises when you're looking at a large tree declaration with relatively complex nesting patterns

You are forced into having this problem with HTML because for some god-forsaken reason HTML lacks the capacity to refer to dom nodes by reference and instead requires you to hand write 35+ layer deep nested trees.

You don't need to have this problem with bsn for the same reason you don't need to have this problem when you write complicated functions - because you can rip chunks of that code out, slap a label on it, and move it somewhere else.

We don't need extra syntactical noise to support people writing code in an unnecessarily garbage manner, we have better tools with which to solve this problem.

[Norlock]

Using Editor GUI as a builtin setting

Since this is something I think of in my own project, I think the Editor should be built in (maybe in an external crate), in Bevy and not as an external app. However you should be able to toggle off for release, and be able to extend/modify bevy's builtin editor:

fn main() {
    let dev_mode = true;

    App::new()
      .set_editor(dev_mode, bevy::Editor::default()) // Just a trait that can returns Bsn
      .set_ui(ui_system) // A system that can return Bsn
      ...
}

fn ui_system(...) -> Bsn {
    if menu_screen {
       return MenuGui::new();
   }  else {
      return GameGui::new();
  }
}

This way the Editor won't hinder anyone that wants to use code only, but can create easy GUI features for the people that want it. And you can make some easy customizations to the bevy editor as well if you want to.

[Type1J]

That could also help in cases that the UI of the editor could be rendered in non-traditional ways, such in VR or AR, since one could add an outer element to the returned BSN with a transform.

[undersquire]

I would say my ideal workflow would be best as a normal Bevy plugin, and would look something like this:

use bevy::prelude::*;

fn main() {
    let mut app = App::new();
    
    app.add_plugins(DefaultPlugins ...);
    
    #[cfg(feature = "my_game_dev_mode")]
    app.add_plugins(bevy::editor::EditorPlugin::default());
    
    ...
    
    app.run();
}

Where Cargo.toml could look like:

[dependencies]
bevy = "0.13"

[features]
default = ["my_game_dev_mode"]
my_game_dev_mode = ["bevy/editor"]

Compiling debug: cargo run
Compiling release: cargo run --release --no-default-features

[Norlock]

Yes the exact implementation I don't really mind. Just the fact that you can easily toggle on and off, I think is a big win. I think one of the big features of Bevy is that is doesn't get in your way, which does happen with some other engines. This design compared to a normal editor will also give you the benefit of easily changing views in the future. A lot of engines are bloated with buttons layouts and ui, but what if you can easily toggle between views.

Like a match statement on button press with (level editor / post fx editor / assets / etc), it is unique and clean.

[afonsolage]

Bevy editor must be an external app, released on package managers and also Steam, like Godot, since many game artists doesn't know how to build or code like that, they usually do small logics inside scripting.

But since Bevy uses composable plugins, there will be a plugin for Bevy editor, or at least, for parts of it, so people can tweak the editor and make their own level editor/custom editor using Bevy editor as base.

[Norlock]

Well you can make an official packaged editor as just a small shell around some basic hello world app with the ability to import/export other repo's. But I can see the trade-off of using a builtin as to having it external, either way if the team feels strongly towards external than that's ok too.

[zimzat]

Reactivity and the VueJS Design

I don't see much mention of VueJS in the thread so I want to highlight a few aspects of their design choices which make a huge impact on usability and assumed functionality compared to React. Sorry if this is otherwise a distraction from the overall Rust / Bevy UI discussion; just some food for thought.

Templates are separate from logic.

Vue moves all template structure to a <template> section, unlike React which allows data, logic, and syntax to be interpolated at will. This separation makes it much easier to compartmentalize context variables, data calls, and method declarations. When the template is at the top of the file it also gives the developer awareness of what to expect the logic of the component to be doing at a glance without having to work backwards. This separation makes it easier to pre-process the template once (apply some optimizations to what gets rendered or only render parts of it based on the dependent data).

Vue's enforcement of Single-File Components is especially helpful to keeping things simple.

State is preserved between calls.

React says "function Component" and everyone hears "function[al] Component". React's function components are the render loop so they have to recompute everything on every call and developers have to go out of their way to avoid it. Vue handles this by providing an explicit "setup" function separate from the various computed values and methods which access the existing state. This removes the complicated const [x, setX] syntax from React and leaves it up to the framework to trigger state changes and listeners. Computed values only get re-computed if the dependent data changes. This is more inline with the way Bevy's Change Detection works.

Slot Content, Named, Scoped https://vuejs.org/guide/components/slots.html

Vue templates provide syntax to inject child components separately from props and for those child components to receive props from the intermediate component. This allows for a better separation of concerns (data vs presentation), removes the need for a "context" component, and provides a way to style child component data from the parent component, making it easier to have generic "Table" type components without having complex prop structures and render functions.

2¢

[nicopap]

Cuicui roadmap

Since my first comment in this thread: #9538 (comment) I implemented templating and did research possible improvements.

I now have a roadmap of features: nicopap/cuicui_layout#111 I'll copy the TOC here, you can follow the link for more details on each individual item:

1. An import system where you can use templates defined in other .chirp files
2. A predefined widget library you can tap from for your own uses.
3. dynamic-query-based components, so that it's possible to declaratively define dynamic lists based on queries.
4. A pre-processed file format. For loading scenes without having to parse .chirp files nor interpreting them.
5. More layout rules.
6. Better error messages, recovering parsers.
7. Bidirectionality. So that it's possible to save back into .chirp files edited scenes. It should preserve comments, whitespaces etc.
8. UI visual editor.

Learnings

I still think I'm moving in the right direction. I'm designing a declarative language with no control flow, which goes against the dioxus, and [JS framework du jour] zeitgeist. But from (1) playing with my framework (2) helping people design and implement two different bevy UI frameworks, one web-tech-based, and one dioxus-based, I still prefer cuicui. The extremely narrow scope of cuicui, and its insistence in leaving state management to the ECS makes it particularly attractive.

There is still a lot of work required to prove that the approach does work, but each passing day, my confidence grows.

[vultix]

I find one of the hardest things to get right in any UI framework is reactivity: data binding, event handling, composability, and managing both local and global state. Additionally, if Bevy is going to offer a first-class UI solution, prioritizing developer ergonomics and code readability is a must.

This comment is an exploration of the design space for incorporating reactivity into @cart's initial scene proposal within the Bevy game engine, focusing on readability and ergonomics. I won't be implementing anything—just exploring the design space.

Important

The intention here is to explore the design space, not to rigorously specify technical details. This means:

• Naming, syntax, and APIs are all tentative and meant to illustrate concepts. I expect every last detail to change, and won't bikeshed
• Details will be glossed over. My intention is primarily to convey ideas
• Some of this may not be feasible, but I'll try to remain grounded

Note

I chose to use the word Widget for UI components, as bevy already heavily uses the term Component in the ECS context.

Scenario 1: A simple counter widget

Project development time and bug count grow quadratically with the number of lines of code in a project. If bevy UI is going to be competitive with other UI frameworks, we are going to need to be similar in code size, readability, and ergonomics to other frameworks.

With this in mind, I'm going to be using Svelte as both inspiration and a baseline to evaluate our designs, as I find it tends to set the bar in terms of developer experience. The svelte counter widget is just 14 lines of simple, self explanatory code. I'm hoping we can get the bevy counter widget to share these properties

Svelte Counter Example

Try it on the Svelte REPL

<script>
  export let count = 0;

  function increment() {
	count += 1;
  }
  function decrement() {
	count -= 1;
  }
</script>

<button on:click={decrement}>-</button>
{count}
<button on:click={increment}>+</button>

Bringing Data Binding to the Bevy Scene Format

// counter.bsn
Div [
    Button {text="-" on:click={decrement}}
    Label {bind:text={count}}
    Button {text="+" on:click={increment}}
]

I've made a few changes to @cart's proposed Bevy scene format here. Notably, I'm employing a=b for property assignments instead of a:b. This allows us to prepend operators like on: for event listeners and bind: for data binding.

The scene file is responsible for describing the node layout of the scene, data bindings, and event bindings. In the future we'll likely want to introduce simple logic in the form of loops, if, else if, and else blocks.

Thoughts / future improvements:

• Is it necessary to wrap everything in a Div tag? I imagine these widgets will always be spawned on an entity with a NodeBundle.
• A related question - when spawning a scene on a given entity, do the components described in the root of the scene file get attached to that entity, or as children of that entity?
• bind:text implies a two-way data binding, this is only a one-way binding
• For string bindings, we may also want to support a format! syntax like bind:text="Count: {count:02}".
• In the rust code, count is going to be an integer. Do we want to do implicit string conversion using the Display trait?

Binding Rust Logic to the Counter Widget

#[derive(Default, Component, Scene, Reflect)]
#[scene(path="counter.bsn")]
struct CounterWidget {
    pub count: isize,
}

#[reflect]
impl CounterWidget {
    fn increment(&mut self, time: Res<Time<Real>>) {
        self.count += 1;
        
        // No real reason for this, other than to demonstrate that event handlers may wish to use system parameters
        println!("Increment clicked at time: {}", time.elapsed())
    }

    fn decrement(&mut self) {
        self.count -= 1;
    }
}

// Setting up the app
app.register_scene::<CounterWidget>();

// Creating an instance of the widget is as simple as spawning an entity with a NodeBundle and a CounterWidget.
app.spawn((
    NodeBundle {...},
    CounterWidget::default()
))

I'm hoping that we can create an API as ergonomic as this, with derive(Scene) largely relying on reflection to setup bindings and event listeners for us. The biggest unknown for me is if we can do method reflection using the #[reflect] attribute, using that information to dynamically create systems. My intent is that at runtime when we parse the counter.bsn file, we can use reflection to say "is there a function with the same name as the asked for event handler or binding? If so, setup a system. If not, raise an error"

If reflection doesn't work, I'm fairly confident we can use a dedicated set of attribute macros to generate the boilerplate.

I do worry that this is a bit too magical. It's not immediately apparent that increment, decrement, and count will be bound to the scene without looking at the associated scene file. In svelte, the logic and view sit right next to each-other, so that's less of a problem.

That said, I love how readable and maintainable this is. It's every bit as short as the svelte implementation, and it's understandable at a glance. I've tried finding less magical equivalents to this using normal systems, and it typically requires much more code to wrap your head around and feels error prone.

Under the Covers

The above example is cool and all, but what exactly is it doing? This is a lower level implementation that doesn't use reflect or macros. I've tried to keep the API as ergonomic as possible, but still feel it's less than ideal. Our "magical" reflect-based or macro-based wrapper will be built on top of this.

First, we know that we'll want a component to represent our widget's local state.

#[derive(Default, Component)]
struct CounterWidget {
    count: isize,
}

impl CounterWidget {
    fn increment(&mut self, time: Res<Time<Real>>) {
        self.count += 1;
        
        // No real reason for this, other than to demonstrate that event handlers may wish to use system parameters
        println!("Increment clicked at time: {}", time.elapsed())
    }

    fn decrement(&mut self) {
        self.count -= 1;
    }
}

The widget is backed by a scene, so we'll implement a Scene trait on it.

impl Scene for CounterWidget {
    fn path() -> &'static str {
        "counter.bsn"
    }

    fn initialize(scene: &mut SceneInit) {
        scene.add_listener("increment", counter_increment);
        scene.add_listener("decrement", counter_decrement);
        scene.add_system(update_counter_on_count_change);
    }
}

fn counter_increment(
    mut event: ListenerMut<ClickEvent>,
    counters: Query<&mut Counter>,
    time: Res<Time<Real>>,
) {
    counters.get_mut(event.entity).increment(time);
}

fn counter_decrement(mut event: ListenerMut<ClickEvent>, counters: Query<&mut Counter>) {
    counters.get_mut(event.entity).decrement();
}

fn update_counter_on_count_change(
    mut counters: Query<(&CounterWidget, &mut SceneRef), Changed<CounterWidget>>,
) {
    for (counter, mut scene_ref) in &mut counters {
        // TODO: This will update the binding if any field on CounterWidget changed. Look into memoization
        scene_ref.update_binding("count", counter.count);
    }
}

Warning

This Scene API is very much a work in progress, and I'd love suggestions for how to improve things. I'm generally a fan of the event listeners, but don't like how the data binding is being done right now.

Implementing Scene defines where the bevy scene file is located, and initializes systems for data binding and event handling. Event listeners are just normal systems, modeled after bevy-eventlistener. For data binding you register normal systems that use ECS change detection to update the scene's bindings using a &mut SceneRef component.

Scene implementations have lots of boilerplate, scaling with the number of event listeners and bindings in your scene, which is exactly why I hope we can generate it using reflection and/or macros.

All that's left now is to register the CounterWidget as a scene and spawn it!

// Setting up the app
app.register_scene::<CounterWidget>();

// Creating an instance of the widget is as simple as spawning an entity with a NodeBundle and a CounterWidget.
app.spawn((
    NodeBundle {...},
    CounterWidget::default()
))

Scenario 2: Coming Soon

I'm going to wait for feedback before diving into another scenario, but I want to continue down this road if everyone likes where it's going. I have plans for scene event dispatchers, nested widgets, two-way data binding, slotted content, and bsn logic (loops, match, and if/else).

Please let me know if you have suggestions or if there's anything else I should look into. Thank you!

[Diddykonga]

Perhaps like this?

// counter.bsn
CounterWidget {count: 0}  [
    Button {text: "-", on::click: decrement},
    Label {text: <= {count} },
    Button {text: "+", on::click: increment},
]

<= for bind::from, => for bind::to.
(I used commas, because I despise white-space languages)

[afonsolage]

Scoped widget

I don't like the idea of attaching Widgets to bsn files, it reminds the my old days of JSF Beans, which doesn't scale very well when you have a lot of Widgets on the same bsn file.

I would think it's best to keep things inside a Widget, like so:

// counter.bsn
Div [
    CounterWidget {count: 0} [
        Button {text="-" on:click={decrement}}
        Label {bind:text={count}}
        Button {text="+" on:click={increment}}
    ]
]

This way, we keep CounterWidget scoped and it is possible to solve it by TypePath reflection. One problem that this may arise is nesting widget conflits, let's see another example:

Div [
    CounterWidget {count: 0} [
        CounterWidget {count: 100} [
            Button {text="-" on:click={decrement}} // which one to refer? inner ou outer widget?
            Label {bind:text={count}}
            Button {text="+" on:click={increment}}
        ]
    ]
]

It could be solved by introducing ids (which was the solution on JSF), using something like CounterWidget {counter: 0, id:"inner"} and now it could just be Button {text="-" on:click={inner.decrement}}

Auto complete

One hard requirement from bsn is DX, so it must play nice with rust-analyzer. Using the add_listener that way, it won't be possible, since this seems to be using some kind of run-time map to map the name of the listener. AFAIK the only way to make RA play nice in this case, is if bsn reaction uses Rust type system, so instead of saving the string, it would have to register some type Trait (pretty much like System<In = In, Out = Out>)

Single File Components

I do worry that this is a bit too magical. It's not immediately apparent that increment, decrement, and count will be bound to the scene without looking at the associated scene file. In svelte, the logic and view sit right next to each-other, so that's less of a problem.

We actually can use SFC (https://vuejs.org/guide/scaling-up/sfc.html) in Bevy, using bsn! macro, hypothetically, it could be something like:

impl Scene for CounterWidget {
    fn initialize(scene: &mut SceneInit) {
        // init code
    }
    
    fn setup(&self) -> Bsn {
        bsn! {
             CounterWidget {count: 0} [
                Button {text="-" on:click={decrement}}
                Label {bind:text={count}}
                Button {text="+" on:click={increment}}
            ]
        }
    }

    fn increment(
        mut event: ListenerMut<ClickEvent>,
        counters: Query<&mut Counter>,
        time: Res<Time<Real>>,
    ) {
        counters.get_mut(event.entity).increment(time);
    }

    fn decrement(mut event: ListenerMut<ClickEvent>, counters: Query<&mut Counter>) {
        counters.get_mut(event.entity).decrement();
    }
    
    fn update_counter_on_count_change(
        mut counters: Query<(&CounterWidget, &mut SceneRef), Changed<CounterWidget>>,
    ) {
        for (counter, mut scene_ref) in &mut counters {
            // TODO: This will update the binding if any field on CounterWidget changed. Look into memoization
            scene_ref.update_binding("count", counter.count);
        }
    }
}

[viridia]

Over the last month I've been doing a lot of prototyping on UI solutions. One thing that has stymied me again and again is that there's a fundamental conflict between React-style reactivity and ECS.

For example, suppose you want to have a React-style "hook" that returns a reference to a resource. Let's use Dioxus-style syntax for this example:

fn example(cx: Cx) {
   let res1 = cx.use_resource::<Res1>();
   let res2 = cx.use_resource_mut::<Res2>();

   // output your UI nodes here
}

This seems simple, but it won't work in Bevy. The reason is because in order for "Cx" to be able to return a mutable reference to a resource, it has to have a mutable borrow on the ECS world. Which in this case produces a borrow conflict. Thus, your UI component can access at most a single mutable resource - which is fine for simple components, but obviously runs in to scaling problems for anything complex.

This would be trivial to solve if, for example, you could pass around a reference to a resource in an Arc<Mutex<>>, but that is not how resources work.

Note that interior mutability of Cx won't save you here, because all of the techniques for interior mutability limit the lifetime of the borrow.

You can also try to do something like dependency injection, like one-shot systems:

fn example(cx: Cx, res1: UseRes<Res1>, res2: UseResMut<Res2>) {
   // output your UI nodes here
}

Here I'm assuming that UseRes is like Res except that it also adds reactive dependency tracking. The problem here is that while this may work for resources, it doesn't work for components or anything that is looked up via a runtime value:

fn example(cx: Cx, cmp: Component<MyComponent(e)>) { // e is not a constant!
   // output your UI nodes here
}

However, even if you manage to solve these problems, there's another problem: event handlers:

fn example(cx: Cx) {
   let res = cx.use_resource_mut::<Res>();
   
   let elt = /* create UI elements */
   elt.on_pointer_down = |ev| {
      /** Do something with 'res' */
   }
}

The problem here is that the event handler (and I mean this generically, whether you are using bevy_mod_picking or some other system) lives longer than the function that creates the UI. (Unless we're talking pure immediate mode, which has other problems). Thus, you can't pass around res because it doesn't live long enough. Also, because (if we're using something like bevy_eventlistener) there's no guarantee that the event handler will be called on the same thread as the code that sets it up.

You can of course design an event handler system in which the handler accesses the resource independently (not using a closure), but now you lose the reactive tracking relationship between the render function and the handler.

Why do I say that there's a "fundamental conflict"? You have to think about what life was like before React. One of the key advancements of React and others like is is the idea that UI components are self-contained and modular. A UI component isn't just about rendering, it's also about state and interaction, and all those things are contained within the component and can be instantiated many times. Unlike, say, Qt or Java Swing, a React component doesn't need to define a bunch of data types and functions at the global level. Any state that it needs can be shared via closures with its event handlers, it all comes together as a package, you don't have to create/register a separate state object every time you want to re-use a widget.

ECS is just the opposite: the ECS philosophy is that everything gets shattered into individual fragments (systems, components, resources) all of which exist at a global level. Complex widgets are no longer self-contained packages. Everything needs to be visible to the scheduler so that it can work its parallelism magic.

Frameworks like Dioxus or Iced don't have these problems because their states are portable: you can pass around state from one widget to another, up and down the hierarchy, without having to pass some mutable borrow of a World.

[nicoburns]

ECS forces us to turn what would otherwise be internal details of a widget into a globally registered system. It means that widgets are no longer "self-contained" but rather atomized into a bunch of global definitions: custom types, custom systems, custom components, setup registration and so on. This is fine if your widgets are simple. The problem comes when you have hierarchies: the behavior of the "OK" button for the item picker dialog is completely different than the behavior of the "OK" button for the quit game dialog. Now you need to invent and register a new ECS component for every different behavior that a button can have.

@viridia I wonder if you could solve this with a single component WidgetState that contains a Box<dyn Any> that is downcast by the framework before being passed to the component? You could probably put a Default bound on it too so that the framework could construct prior to the first render. This would then be accessed by the widget using something like a State<&mut MyStateType> parameter.

[IronGremlin]

Now you need to invent and register a new ECS component for every different behavior that a button can have.

Dumb question maybe but if this paradigm scales perfectly fine for writing a whole ****ing video game why are we convinced that it will not scale well for writing some UI for that video game?

[viridia]

@IronGremlin My previous game + editor, written in TypeScript, consists of 70k lines of code. Scanning the source code, there are over 3,000 places where a UI component is instantiated.

Now imagine that every widget takes 5x as much effort as before. This isn't a problem for a game, because game UIs generally aren't that complex. It's a big problem for tools, though.

[IronGremlin]

But that is not what you're talking about.

You are talking about special case behavior dispatch and managing the state of related entities.

Neither of those have anything to do with the lines of code in your project or the number of times you instantiated a widget.

For special case behavior dispatch, we just got one-off systems, and for managing the state of related entities, you store the state in components on those entities, and then you query for them. If you have common states or structures with minor variations typically you make a custom command or a function.

Like I might get behind the idea that some of these solutions aren't especially ergonomic for UI development because X, but it kind of feels like we aren't even trying to use the same problem solving toolset we'd use for a complex ECS problem, it feels like we're trying to use the same problem solving toolset we'd use for a complex web UI - which is confusing because this isn't a web UI, we don't have the same primitives, and I shudder to think that anyone would actually WANT them.

[CooCooCaCha]

I’d also like to remind that this isn’t just UI. It’s a new scene/prefab system with reactivity.

[vultix]

For a while now I've been considering if adding for-each systems may be a big ergonomics boost, specifically for event handling and bsn!

What are for-each systems?

Today, a typical bevy system may look like this:

fn apply_velocity(
    mut query: Query<(&mut Transform, &Velocity), Without<Disabled>>,
    time: Res<Time>,
) {
    for (mut transform, velocity) in &mut query {
        transform.translation.x += velocity.x * time.delta_seconds();
        transform.translation.y += velocity.y * time.delta_seconds();
    }
}

// Register the system
app.add_systems(FixedUpdate, apply_velocity)

A for-each system automatically applies the for loop for you:

fn apply_velocity(transform: &mut Transform, velocity: &Velocity, time: Res<Time>) {
    transform.translation.x += velocity.x * time.delta_seconds();
    transform.translation.y += velocity.y * time.delta_seconds();
}

// Register the system
app.add_systems(FixedUpdate, apply_velocity.for_each::<Without<Disabled>())

Additionally, there should be some way to run a for-each system on a single entity:

world.run_system(apply_velocity.on_entity(my_entity))

How does this improve event handling ergonomics?

Currently, in event handlers, we have to write a query to modify or read components of the entity handling the event. This feels unintuitive and unergonomic, as we already know there is exactly one entity handling the event, meaning we need to do query.get(entity).unwrap(). A for-each system would allow us to simplify this.

Example: bevy_eventlistener goblin example

Consider the minimal example of the excellent bevy_eventlistener library. It spawns a "Goblin" entity with 3 pieces of armor.

commands
        .spawn((
            Name::new("Goblin"),
            HitPoints(50),
            On::<Attack>::run(take_damage),
        ))
        .with_children(|parent| {
            parent.spawn((
                Name::new("Helmet"),
                Armor(5),
                On::<Attack>::run(block_attack),
            ));
            parent.spawn((
                Name::new("Socks"),
                Armor(10),
                On::<Attack>::run(block_attack),
            ));
            parent.spawn((
                Name::new("Shirt"),
                Armor(15),
                On::<Attack>::run(block_attack),
            ));
        });

Each piece of armor can receive an "Attack" event. The armor reduces the damage of the attack, potentially preventing the event from propagating up to the Goblin.

/// A callback placed on [`Armor`], checking if it absorbed all the [`Attack`] damage.
fn block_attack(mut attack: ListenerMut<Attack>, armor: Query<(&Armor, &Name)>) {
    let (armor, armor_name) = armor.get(attack.target).unwrap();
    // Handle the attack
}

/// A callback on the armor wearer, triggered when a piece of armor is not able to block an attack,
/// or the wearer is attacked directly.
fn take_damage(
    attack: Listener<Attack>,
    mut hp: Query<(&mut HitPoints, &Name)>,
    mut commands: Commands,
    mut app_exit: EventWriter<bevy::app::AppExit>,
) {
    let (mut hp, name) = hp.get_mut(attack.listener()).unwrap();
    // Take damage, subtracting health from hp
}

Notice that in both event handlers, we need to do query.get[_mut](handler_entity).unwrap(). I find it to pretty much always be the case that a handler system needs to extract or modify information of the entity handling the event. There's only ever one such entity, so a for-each system running on just that entity simplifies things:

fn block_attack(armor: &Armor, name: &Name, mut attack: ListenerMut<Attack>) {
    // Handle the attack
}

fn take_damage(
    hp: &mut HitPoints,
    name: &Name,
    attack: Listener<Attack>,
    mut commands: Commands,
    mut app_exit: EventWriter<bevy::app::AppExit>,
) {
    // Take damage, subtracting health from hp
}

Example 2: A Counter UI widget

The place where I think this really shines is when using bsn! to build scenes and UI components.

Here's what a simple Counter UI widget might look like without for-each systems:

#[derive(Default, Component)]
struct CounterWidget {
    pub count: isize,
}

fn increment_counter(mut query: Query<&mut CounterWidget>, event: Listener<Click>, time: Res<Time<Real>>) {
    let mut widget = query.get_mut(event.listener()).unwrap();
    widget.count += 1;

    // No real reason for this, other than to demonstrate that event handlers may wish to use system parameters
    println!("Increment clicked at time: {}", time.elapsed())
}

fn decrement_counter(mut query: Query<&mut CounterWidget>, event: Listener<Click>) {
    let mut widget = query.get_mut(event.listener()).unwrap();
    widget.count -= 1;
}

impl Scene for CounterWidget {
    fn setup() -> Bsn {
        bsn![
            Button {text="-" on:click={decrement_counter}}
            Label {bind:text={CounterWidget.count}}
            Button {text="+" on:click={increment_counter}}
        ]
    }
}

This isn't terribly, but feels clunky. The actions of incrementing and decrementing feel decoupled from the widget itself, and there's a lot of extra code just to get access to the widget handling the events.

Here's what the same example could look like with for-each systems:

#[derive(Default, Component)]
struct CounterWidget {
    pub count: isize,
}

impl CounterWidget {
    fn increment(&mut self, time: Res<Time<Real>>) {
        self.count += 1;
        
        // No real reason for this, other than to demonstrate that event handlers may wish to use system parameters
        println!("Increment clicked at time: {}", time.elapsed())
    }

    fn decrement(&mut self) {
        self.count -= 1;
    }
}

impl Scene for CounterWidget {
    fn setup() -> Bsn {
        bsn![
            Button {text="-" on:click={CounterWidget::decrement}}
            Label {bind:text={CounterWidget.count}}
            Button {text="+" on:click={CounterWidget::increment}}
        ]
    }
}

The magic here is that Counter::increment and Counter::decrement are valid for-each systems! This gives us unparalleled ergonomics for very common use cases.

Note

I imagine we'd support both &mut self and mut self: Mut<Self> in for-each systems, the former automatically marking the component as changed.

Does this make introducing for-each systems worth it?

Bevy used to have for-each systems and they were removed for good reason. I want to revisit those reasons and consider how this may change things.

1. For-each systems were fundamentally limited in a number of ways. They couldn't iterate removed components, filter, control iteration, or use multiple queries at the same time. This meant they needed to be converted to "query systems" as soon as those features were needed.

I'm hoping we can lift some of these limitations. Specifically, we'd need a way for for-each systems to have other queries, making things like this possible:

impl CounterWidget {
    fn increment(&mut self, event: Listener<Click>, buttons: Query<&Name, With<Button>) {
        self.count += 1;

        // A bit contrived, but shows potential use cases
        let button_name = buttons.get(event.target).unwrap();
        println!("Button {button_name} clicked!")
    }
}

2. Bevy should generally have "one way to do things". For-each systems were a slightly more ergonomic way to define a small subset of system types. This forced people to make a "design decision" when they shouldn't need to. It also made examples and tutorials inconsistent according to people's preferences for one or the other.

This argument definitely still applies, to such an extent that I'd very much consider disallowing for-each systems anywhere except event listeners like this.

3. There were a number of "gotchas" for newcomers that constantly come up in our support forums and confused newbies:
   • users expect &mut T queries to work in foreach systems (ex: fn system(a: &mut A) {}). These can't work because we require Mut<T> tracking pointers to ensure change tracking always works as expected. The equivalent Query<&mut A> works because we can return the tracking pointer when iterating the Query.
   • A "run this for-each system on some criteria" bug that was common enough that we had to cover it in the Bevy Book.

In the case of event handlers, I find it very important that &mut T queries still work, and always mark the component as changed. If we can get that to work I'd consider this an acceptable tradeoff.

4. They increased compile times. Removing for-each systems saved me about ~5 seconds on clean Bevy compiles)
5. Their internal implementation required a complicated macro. This affected maintainability.

I'm unsure to what extent these still apply. At the very least this will be more code to maintain, meaning more bugs, slower compilation time, and larger maintenance burden.

Conclusion

I don't yet have enough data to know if this is a worthwhile tradeoff. I'd love to hear everyone's thoughts on the idea!

More examples of real-world event handling would also be great. I'm curious to see if there'd be considerable improvements in these situations.

[Diddykonga]

What if the event handlers/Listener<T> were provided an EntityMut, giving you direct access to the Entity in question, no longer having a need for extra Queries, unless you wanted them for other purposes.

[gamedev-pnc]

If inheriting from multiple scenes, properties should "cascade" across scenes in the order the scenes are referenced

Hi! Interesting discussion, but I'm confused by the idea of multiple inheritance of scenes. Wouldn't this complicate the scene analysis? In this approach: object's property can be A) cascaded from one of the upper objects in the scene tree or B) inherited from one of the parent scenes, and all those sources can also inherit recursively. It's important to minimize possible count of sources, so initial value assignment to the property could be easily found (instead of overriding it in the child object - it often becomes a mess on practice).

There are reasons why modern programming languages refuse multiple inheritance.

Worth mentioning that style and structure separation (i.e., CSS vs. clean HTML) is very good practice.

[Type1J]

Scene inheritance is more like composition than typical inheritance.

[Diddykonga]

Multiple inheritance/inheritance in general wont be introduced until we get Relations, and implement the IsA Relation, which is a whole other can of worms further down the road.

[mrgzi]

Does anyone interested in implementing Ribir to Bevy? They have their own WGPU backend and there are ready-made widgets. I think it might fit Bevy well.

[lambdadeltakay]

I think the issue is that ribir is a ui library and BSN is supposed to cover prefab entities too I think

[mrgzi]

Can you elaborate? How JMS55 integrated bevy and dioxus then? https://github.com/JMS55/bevy_dioxus

[nicoburns]

Dioxus has an abstracted dioxus-core layer that can be used with any widget backend. It's possible that Ribir may have something similar, but it isn't mentioned in the readme and I suspect it may not as most rust ui frameworks do not.

[Veritius]

I'm making a multiplayer game that has both a dedicated server and client application. The client has the full Bevy rendering pipeline and such, but the dedicated client doesn't - the feature is disabled.

I want to be able to use the same scene file for both applications. If I had to maintain two, it'd be a massive pain, since it'd be a larger workload and introduces potential issues where each app has different components in the same scene.

I'll give this example in some pseudo-YAML definition.

components:
     - type: Handle<Mesh>
       path: "../mdls/hello.obj"
     - type: Transform

On the client, both the mesh handle and transform components would be loaded. But on the server, since it doesn't even know what a mesh is, I want it to simply ignore the Handle<Mesh> component.

[matthunz]

Hey everyone 👋 I have proposal in my crate bevy-compose for reactivity using the react/Xilem pattern. This uses a "medium-grain" reactive approach that has a coarse reactive system like react, but finer control with Arc::make_mut like Xilem.

Counter app example:

fn ui(count: Res<Count>) -> impl Compose {
    flex((
        format!("High five count: {}", count.0),
        flex("Up high!").on_click(|mut count: ResMut<Count>| count.0 += 1),
        flex("Down low!").on_click(|mut count: ResMut<Count>| count.0 -= 1),
    ))
}

More advanced counter example, using lazy to create a component:

fn ui() -> impl Compose {
    lazy(|count: Res<Count>| {
        memo(
            count.0,
            flex((
                format!("High five count: {}", count.0),
                flex("Up high!")
                    .on_click(|mut count: ResMut<Count>| count.0 += 1)
                    .on_hover(|| {
                        dbg!("Hover!");
                    }),
                flex("Down low!").on_click(|mut count: ResMut<Count>| count.0 -= 1),
                if count.0 == 2 {
                    Some("The number 2!")
                } else {
                    None
                },
            )),
        )
    })
}

[aschrijver]

I am not in the loop with Bevy project state and see a lotta great evolution going here. I just wanted to mention recently watching a jaw-dropping presentation by Rik Arends of Makepad who create this whole UI framework in Rust where UI is in a Figma-like format you can adapt at runtime. And they also built this great editor for doing so.. using Makepad: https://makepad.dev

[viridia]

Since I last posted here, I've been doing a ton of Bevy UI-related work (R&D mostly), written a lot of code, and learned many things. My earlier comments in this thread seem naive to me now :) If could get readers in this thread to take away just one idea, it would be the need to solve problems in the right order: don't build the penthouse before you've laid the foundation.

More specifically, I would want to move the focus away from both widgets and template syntax:

• Implementing widgets at this point is premature - until we have a framework for declarative updates (explained below), widgets are just technical debt.
• Template syntax is a "nice to have" - I've demonstrated to my own satisfaction that pretty much any UI you can build with a template, you can also do with fluent syntax or builder patterns. While it might not be quite as succinct, the advantage is that these patterns can evolve rapidly as your UI framework develops. This is important so that you don't get locked in to a particular design approach too early.

The really hard problem to solve is "declarative incremental updating", a term which I will need to unpack:

• "declarative" in the sense that we can define our UI hierarchy declaratively (rather than having to imperatively write "add child widget to parent")
• "incremental" in the sense that we're not re-constructing the UI every time there's a change, we're able to apply patches to the UI hierarchy based on changes in app state.
• "updating" in this case refers to the fact that the same code is used for both the initial creation of the hierarchy, and for applying changes to it. This is critical, because it's the key productivity win for UIs in the last 20 years. It's why React/SwiftUI/Elm/Svelte/Vue and all the rest are superior to Java AWT and MFC.

The key reason why bevy_ui as it exists today is so tedious and unergonomic is not because it lacks a templating language, but because creation and updating are separate activities. When these activities are split, it means that the developer has to fetch and carry a lot of state between them, which in turn entails a lot of extra coding. By itself, a templating/macro language won't solve this problem, and can potentially make it worse unless the templating language is built on top of a robust updating solution.

Unfortunately, as I have found, an incremental updating framework can't be just bolted on to ECS, at least not in a straightforward way. "Widgets" can't be "systems" because the runtime scheduling is completely different. Widgets also can't be systems because the same widget appears in many places, whereas re-registering a system doesn't cause it to get run multiple times.

An updating solution may or may not be "reactive": immediate-mode systems like egui are both declarative and unify construction and updating, but they are not reactive in the sense of React, Solid or MobX.

In the last six months I've built two different reactive UI frameworks for bevy: quill and bevy_reactor. The former was a coarse-grained approach modeled after React and Dioxus, and latter a fine-grained approach modeled after Solid. I've convinced myself that I can build a UI framework that's good enough to build an editor; however, I also recognize that my approach has a number of weaknesses that I can't solve by myself.

One weakness is that these frameworks operate in exclusive mode. I know we've discussed this before, and while the general feedback I've gotten is that it's OK for a UI widget to require world access, I'm not sold on this. My reason has nothing to do with performance, but simply because coding anything complicated with exclusive world access means you have a lot more problems with the Rust borrow checker. It means that relatively simple algorithms turn into puzzles.

The other problem is that even though my framework is built on top of ECS, the style of coding required to use it is not very ECS-like. That is, after all the effort we've spent educating people how to rethink their game architecture in terms of entities and components, then we get to UI and say "Oh, and you'll have to learn an entirely new approach when it comes to doing UI."

Part of the tension here is due to a mismatch between what UI needs, and what ECS provides. For example:

• Systems and queries are optimized to operate on relatively flat data, or at least data that can be processed homogeneously, whereas UI hierarchies tend to be heterogenous and have deep hierarchies;
• Systems allow representing state either completely globally (entities and resources) or strictly local (the Local injection), but UIs generally need "semi-local" state, that is, state that is shared between a parent widget and its children, but neither global nor unique (in the sense that there may be multiple constellations of the same parent/child type, each with their own semi-local state).
• Systems allow injection of arbitrary data from the world, but only if the type is known at compile time, and only at the top level (meaning that if you have a system that calls other functions, injection only happens at the top); A complex dialog box or preferences panel won't always know what resources and entities are being used by it's children, so it can't inject what is needed without de-encapsulating the entire widget hierarchy - that is, making the state dependencies of every leaf widget visible to the root of the tree.

The question I keep asking myself is, is it possible to build a widget that looks more like a system? One which supports dependency injection, non-exclusive access, and so on? And every time I start to think about this I run into a wall.

In any case, what I have now is a framework that is powerful, in the sense that a small number of lines of code lets you do a lot. In that sense it's productive and ergonomic. However, it's also occasionally tricky and surprising, sometimes you have to jump through odd hoops to solve what ought to be a straightforward problem.

(I should mention as an aside that while I have doubts about parts of my framework, there are other parts that I am completely happy with, such as style builders. More generally, it's mainly the small, inner reactive kernel that troubles me; most of what I have built on top of it I am fine with.)

[viridia]

Small correction: adding systems to a schedule multiple times will in fact cause them to be run multiple times every time the schedule is run.

Thanks, that's good to know.

I said in my post that "widgets cannot be systems", but in in bevy_egui that's exactly what widgets are: they run every frame and reconstruct the UI each time. This approach gives you a lot of the benefit of systems:

• Being able to inject arbitrary parts of the world
• Being able to have separate lifetimes for injected data so that you can access multiple entities and resources without borrow checker issues.
• Being able to use queries and query filters are part of your UI.
• Being able to react to changes in data without complex dependency tracking.

Unfortunately, you also lose a lot with the immediate mode approach:

• The fact that you're rebuilding the whole UI every frame is expensive if your UI is complex.
• A lot of sophisticated layout algorithms (including flexboxes and grids) requires two layout passes, which is impossible in immediate mode.
• Doing "semi-local state" gets complicated.
• Immediate mode isn't really compatible with bevy_ui, which is by nature a "retained" scene graph.

The question I keep asking myself is, is there a way to have something like an ECS system which both creates and updates a hierarchy? It would have to do more than just that, though:

• To support both creation and update, it would have to have a persistent object that keeps track of what's already been created, and then compute the diff between old state and new.
• These systems would themselves have to be hierarchical, meaning "systems within systems", where each sub-system had its own dependencies and injections. Otherwise the root of your UI has to inject every bit of data needed by every sub-panel and every lead widget.
• You'd want sub-panels to be able to update independently on their own schedule, so you're not rebuilding the entire UI just because one checkbox changed from checked to unchecked state.
• You'd want a way to inject or create "local state" but in a way that could be passed to child systems.
• What decides when an update happens? Do we have to manually write my_resource.is_changed() and then manually call my_template.update()? How does that work for queries?

At this point, it seems like the challenge ball is in the ECS court: in other words, you can't solve this by building a better UI framework on top of ECS, you'd actually need to extend ECS itself to accomplish this.

[matthunz]

Really interesting points 👀 totally agree here!

I'm curious how well the new Bevy observers will ease this problem. In my experience making a UI widget with components like FontSize and having the widget react to changes has been pretty straightforward already. I feel like maybe with observers we'll finally have an easier way of working with linear-style reactivity, rather than in parallel

[JMS55]

Very much agree with your list of problems and questions @viridia. A lot of these are much the same issues I brought up 10 months ago in a different thread in this discussion. I think it's valuable to read that over again.

These are also the same problems (minus UI <-> ECS interaction) that other UI frameworks are solving or have already solved, both Rust-based and in other languages. I know it's contentious, but I'd rather start with a reactive UI system that reuses an existing UI framework, rather than programming our own with much less developer time spent on it (given all the other parts of a game engine we spend time on). Likely Dioxus or Leptos. More asset/art based, less technically complex UI systems could be built with an alternate crate focused around hot-reloading BSN assets.

For UI <-> ECS interactions, I do think exclusive world access makes sense. Sure it's not "clean", but it honestly works well enough. It's the pattern I follow in bevy_dioxus, and I'm happy enough with it.

[dannymcgee]

I've also been working a lot with Bevy UI lately (including user-interactable 3D elements, which is something that's not often considered part of the "UI" question but does suffer from a lot of the same pain points). I broadly agree with your assessment here, but I do have one point of contention, and a couple of patterns I've found that may be helpful.

The key reason why bevy_ui as it exists today is so tedious and unergonomic is not because it lacks a templating language, but because creation and updating are separate activities.

I think I would actually flip the priority order here, but for largely the same reasons you point out. The biggest difficulty I have defining re-usable UI abstractions currently is the fact that describing a UI hierarchy with more than one level of depth requires access to the Commands buffer (and often other resources as well, like ResMut<Assets<_>> and Res<AssetServer> for fonts, materials and the like). That means that your widget-spawning functions need to be systems, which means that you need exclusive &mut World access to invoke them.

In my mind, having a way to statically describe a hierarchy without needing access to those resources is kind of the foundational problem — solving that would go quite a long way toward making UI immediately less painful, and reactive systems could be layered on top of it. That is basically the problem that's solved by the BSN format (and the Scene system more generally).

Systems allow representing state either completely globally (entities and resources) or strictly local (the Local injection), but UIs generally need "semi-local" state, that is, state that is shared between a parent widget and its children, but neither global nor unique (in the sense that there may be multiple constellations of the same parent/child type, each with their own semi-local state).

100% agreed, though I don't think this is necessarily an intractable problem for ECS. I'm sure this isn't a new idea to you, but a pattern I've found myself using a lot is defining a widget-specific State component that lives at the root of a particular widget's hierarchy, and using Query<&Parent>::iter_ancestors and Query<&Children>::iter_descendants to look up and act upon those state components in Update or event-listener systems.

I built a basic checkbox widget recently which is composed of three entities:

(
  CheckboxField,
  CheckboxState(Interaction),
  On<Pointer<Click>>(on_checkbox_click),
  Children([
    (Checkbox(bool), /* various visual stuff */),
    (CheckboxLabel, Text),
  ])
)

The on_checkbox_click system is defined like this:

fn on_checkbox_click(
    event: Listener<Pointer<Click>>,
    mut ew_changes: EventWriter<CheckboxChange>,
    mut q_checkbox: Query<&mut Checkbox>,
    q_children: Query<&Children>,
    q_parents: Query<&Parent>,
    q_root: Query<Entity, With<CheckboxField>>,
) {
    let e_checkbox = if q_checkbox.contains(event.target) {
        Some(event.target)
    } else {
        let e_root = if q_root.contains(event.target) {
            Some(event.target)
        } else {
            q_parents
                .iter_ancestors(event.target)
                .find(|e| q_root.contains(*e))
        };

        let Some(e_root) = e_root else {
            bevy::log::error!("Failed to find Checkbox Field root");
            return;
        };

        q_children
            .iter_descendants(e_root)
            .find(|e| q_checkbox.contains(*e))
    };

    if let Some(e_checkbox) = e_checkbox {
        let mut checked = q_checkbox.get_mut(e_checkbox).unwrap();
        **checked ^= true;
        ew_changes.send(CheckboxChange {
            target: e_checkbox,
            checked: **checked,
        });
    } else {
        bevy::log::error!("Failed to find Checkbox")
    }
}

The CheckboxState(Interaction) stores the highest-priority of the three individual Interaction states so that all three entities can be treated as hovered/pressed if any of them are hovered/pressed. The system for managing that is so long I don't even want to paste it here.

That said, I don't think it would be a huge stretch to design some kind of "hierarchical dependency injection" abstraction that allows for easy and efficient lookups of the nearest instance of a given component among the ancestors of a given entity. This would be similar in practice to React's Context API. You could even write an abstraction like that today with a custom SystemParam, though it wouldn't solve the dubious performance characteristics, nor the fact that, architecturally, this is basically object-oriented programming with extra steps.

[alice-i-cecile]

flecs has good tools for "hierarchical dependency injection". Making that fast and easy is a big part of the value in relations.