Polyphony ECS GUI and future

[dumblob]

Hi @traffaillac , I've discovered your gem "Polyphony" and found myself quite enlightened reading the paper.

Do you have any plans to develop it further? Or is there any successor? Or do you know of someone basing some apps or libraries on Polyphony (or principles and findings found therein)?

[traffaillac]

Hi @dumblob, thanks for your interest in our work! (counting my PhD advisor)
I've always wanted to push this track further but it is very exploratory thus takes a lot of work. I am not aware of any successor or library taking inspiration from it, but if someone does work on such endeavour it should take quite some time. ECS is a radical shift away from OOP and Polyphony just scratched the surface. You should not expect a production-ready ECS GUI framework happening anytime soon, except if a company with enough workforce is willing to invest in it (hopes going to Unity...).
If you are interested in working on a successor I'd be more than happy to collaborate with ideas/code. You may want to check out section 3.3.2 (p136) of my PhD thesis (https://pepite-depot.univ-lille.fr/LIBRE/EDSPI/2019/50376-2019-Raffaillac.pdf), which enumerates the basic operations to support. It is in French but nothing deepl translator couldn't handle ;)

[dumblob]

I apologize for accidentally giving attribution just to you and not to others (incl. your PhD advisor). Hereby I'm rectifying it.

I think the first step to further development would be to release Polyphony under some permissive license (MIT, BSD), so that commercial companies (incl. Unity) can freely use it and adjust/rewrite & extend it as they see fit.

Have you already contacted Unity to take a look at Polyphony?

I'd be more than happy to collaborate with ideas/code.

That's awesome! That'll help a lot.

I'm currently contemplating with many others how (G)UI in Bevy (probably the most advanced ECS Rust full-featured game framework & engine) could work. But there are really many questions to that.

One of the biggest "obstacles" (questions) I see is that there will always inevitably be the need to interface with long-running "continuous" processes (maybe because of networking with a much different frequency than the one ECS pipeline of systems is being executed with, maybe because devs will want to use an already written external library spawing own threads without any poll-like API, maybe because there will be a separate event loop system which will not easily integrate with the ECS event loop, etc.).

A naive "solution" could be to encapsulate each of these "metasystems" into their own thread and then send events between them. We would have ECS metasystem (both 3D combined with ECS GUI in one event loop and thus both in the same thread instance), then some library thread metasystem, then some network thread metasystem, then maybe some input devices thread metasystem, etc. They could communicate with each other like actors (i.e. each would have a queue of events/messages/triggers to work through). The ECS thread would of course additionally use some other threads from the thread pool to distribute computation of some "Systems".

But that sounds too naive - in practice this is yet another thing to research...

Let's keep this thread open for now if you don't mind.

[traffaillac]

I understand that a non BSD-like licenses tends to categorize libraries as "evil" and to be avoided. The fact is my past employer owns the code, and I couldn't enforce a more permissive license (https://gitlab.inria.fr/Loki/PolyphonyECS/-/blob/master/LICENSE). It basically says "free for non commercial use".

Anyway I think Polyphony should not be used per se. It was not meant to be stable, only to test some ideas. If I were to change anything I would recode it from scratch, in particular :
_ switching to Python which has a much better ffi and library ecosystem (NodeJS was a PAIN)
_ using a higher-level rendering backend than OpenGL, but still low enough to control display lists and get the most of ECS Systems (something like bgfx or ANGLE)
_ finding a proper way to store temporary components
_ and many other minor fixes...

Have you already contacted Unity to take a look at Polyphony?

Nope, I was shy but will do soon I promise :3

As for the interface between parallel processes, I think Entities/Components should be used as this interface. This is what Unity is/was doing as far as I remember (jobs that might run in parallel and read/write fences in between). So it is OK to put Systems in threads as long as they use Entities as mediums for their communication. In particular I think one should avoid per-System event queues, because this is introducing another way to share data than Entities+Components. We introduced temporary Components and device Entities exactly for that, to offer an alternative to event queues.

I should underline that my PhD was made in the context of people trying to "hack" existing applications for research, so a core goal of Polyphony was to be simple inside, i.e. all data belongs to Entities, and all code belongs to Systems (no lambdas allowed). In addition, it is easy to patch the application at runtime, e.g. adding a System that alters the mouse position before higher-level Systems get to execute. The inline and predictable execution of Systems is important here. I think a multi-threaded scheduler (like Unity does) "depowers" programmers, in that they do not have direct control over when Systems execute. You would declare on which Components your System depends or after which other Systems it should execute, but then you have no clue whether the overall application will behave the way you intended. If you were to introduce threaded Systems, I think it should be of prime importance that programmers have the most direct control over them. It could be for example inspired by Java Swing (3 threads, "main application thread", "toolkit thread", "event dispatching thread"), i.e. having 3 chains of Systems running in parallel, each of which have a fixed frequency and Systems order.

Now a problem with Entities being used as intermediate communication is if you have a tight coupling between Systems. The classic example in our paper is DepthUpdateSystem (1st System in the whole chain), which does a DFS exploration of the scene tree(s) and assigns a depth Component to each node Entity. If a System down the chain were to change the scene tree, you would have to wait until the next execution of the whole chain to run DepthUpdateSystem again and see an update in the depth Components. This is a BIG problem of ECS, that is you can't say "if this Component changes then run this code now". In (G)UI research this is the basis for dataflow and reactive frameworks, but they never managed to demonstrate the need to apply this principle to entire architectures. In the case of Polyphony, it means that some Component changes might result in a visual change after one or a few iterations of the whole Systems chain, i.e. at most 1/10th of a second. So far I've tried to avoid introducing "special" Systems, e.g. that could execute many times per chain whenever some Components change, because that would make the whole concept more complex for a few specific cases. I think it will depend on how many more situations will happen.

In the end it really depends on the kinds of UI you want to build. A mistake of Polyphony is that we did not really define the context of use beforehand. Being of general purpose makes it really hard to take into account all possible uses of a framework, and it slows down the work very much. What are the kinds of UIs you intend to build ? In-game HUD ? game debugging UIs ? smartphone apps ? desktop apps ? browser apps ? microcontroller/Arduino stuff ?

Cheers,
Thibault

[dumblob]

I understand that a non BSD-like licenses tends to categorize libraries as "evil" and to be avoided. The fact is my past employer owns the code, and I couldn't enforce a more permissive license (https://gitlab.inria.fr/Loki/PolyphonyECS/-/blob/master/LICENSE). It basically says "free for non commercial use".

Hm, this basically means involvement on many other levels than just on the engineering technical level. And that's usually not acceptable. In other words, the pure "price of the product" is not important, but the involvement of management, purchasing/procurement department, taxes (cross-country!), etc. It's just a huge concrete wall to painfully and slowly climb before an engineer can finally touch the code and do the thing.

Anyway I think Polyphony should not be used per se. It was not meant to be stable, only to test some ideas. If I were to change anything I would recode it from scratch, in particular :
_ switching to Python which has a much better ffi and library ecosystem (NodeJS was a PAIN)

That's an interesting observation. I was under the impression that the reason you chose NodeJS was actually easier development. But it wasn't 😉.

_ using a higher-level rendering backend than OpenGL, but still low enough to control display lists and get the most of ECS Systems (something like bgfx or ANGLE)

My personal goal is actually a bit different. Namely to use the ECS GUI architecture with 2D CPU-rendered libraries (I aim for Blend2D specifically - I know the dev for ~12 years already).

Nope, I was shy but will do soon I promise :3

Let me know (e.g. in this thread) how it goes. I'm pretty interested in what will happen then.

As for the interface between parallel processes, I think Entities/Components should be used as this interface. This is what Unity is/was doing as far as I remember (jobs that might run in parallel and read/write fences in between). So it is OK to put Systems in threads as long as they use Entities as mediums for their communication. In particular I think one should avoid per-System event queues, because this is introducing another way to share data than Entities+Components. We introduced temporary Components and device Entities exactly for that, to offer an alternative to event queues.

Well, my worry is how to tackle the following problem.

A company bought a media player software (hereafter referred to as metasystem 1) to embed its visual rendering into their new shiny GUI software S which doesn't exist yet. They also bought an optimized streaming software without any visual part (hereafter referred to as metasystem 2) to embed into S and a proprietary stock-interaction software with its visual rendering (hereafter referred to as metasystem 3) to embed into S too.

All these bought software pieces provide a session-oriented blocking interface. The player SW and stock SW will be controlled from S but additionally independently from outside (so this communication can't go through S).

How would this be mapped to ECS GUI (hereafter referred to as metasystem 4)? From my understanding ideally these all would be modeled as input (identically as keyboard, mouse, etc. already are in Polyphony) and non-blocking output (the "non-blocking" is important otherwise the whole ECS pipeline would get stuck). So inevitably the ECS pipeline has to run concurrently to these 3 embedded pieces of SW. And this is usually best solved by sending a message to a shared buffer.

I should underline that my PhD was made in the context of people trying to "hack" existing applications for research, so a core goal of Polyphony was to be simple inside, i.e. all data belongs to Entities, and all code belongs to Systems (no lambdas allowed). In addition, it is easy to patch the application at runtime, e.g. adding a System that alters the mouse position before higher-level Systems get to execute. The inline and predictable execution of Systems is important here. I think a multi-threaded scheduler (like Unity does) "depowers" programmers, in that they do not have direct control over when Systems execute. You would declare on which Components your System depends or after which other Systems it should execute, but then you have no clue whether the overall application will behave the way you intended. If you were to introduce threaded Systems, I think it should be of prime importance that programmers have the most direct control over them.

Exactly. That's why I don't want to interleave any of the metasystems above, but rather let them communicate in a disciplined highly constrained manner through input & output buffers. This can be modelled as actors or as signal-slot or as pub-sub or as channels from Go lang, etc.

Now a problem with Entities being used as intermediate communication is if you have a tight coupling between Systems. The classic example in our paper is DepthUpdateSystem (1st System in the whole chain), which does a DFS exploration of the scene tree(s) and assigns a depth Component to each node Entity. If a System down the chain were to change the scene tree, you would have to wait until the next execution of the whole chain to run DepthUpdateSystem again and see an update in the depth Components. This is a BIG problem of ECS, that is you can't say "if this Component changes then run this code now". In (G)UI research this is the basis for dataflow and reactive frameworks, but they never managed to demonstrate the need to apply this principle to entire architectures. In the case of Polyphony, it means that some Component changes might result in a visual change after one or a few iterations of the whole Systems chain, i.e. at most 1/10th of a second. So far I've tried to avoid introducing "special" Systems, e.g. that could execute many times per chain whenever some Components change, because that would make the whole concept more complex for a few specific cases. I think it will depend on how many more situations will happen.

Yep, this was one of the most enlightening parts! But I feel somehow that all these (G)UI cases can be abstracted into a finite number of such Systems (maybe just few dozen) and thus once this'll be researched, there will be not much need to extend it. I'd call it a "first normal form".

In the end it really depends on the kinds of UI you want to build. A mistake of Polyphony is that we did not really define the context of use beforehand. Being of general purpose makes it really hard to take into account all possible uses of a framework, and it slows down the work very much. What are the kinds of UIs you intend to build ? In-game HUD ? game debugging UIs ? smartphone apps ? desktop apps ? browser apps ? microcontroller/Arduino stuff ?

My grandiose intent (TM) would be to have a competitor to Qt 😆. I.e. desktop/tablet/smartphone/web interactive framework. Ideally with bindings to several languages (to avoid dependence on any particular language features).

On the other hand, the core of ECS GUI feels pretty much like an abstract concept fully independent from any implementation. So maybe a detailed architecture standard (imagine CSS, HTML, ...) covering whole ECS GUI could also be the way to go (but such a standard can't be developed without having a working protope beforehand). The research paper (not your thesis) about Polyphony could be also viewed as such a standard 😉.

[traffaillac]

That's an interesting observation. I was under the impression that the reason you chose NodeJS was actually easier development. But it wasn't 😉.

Prior to JS, I experimented with ECS on C, Smalltalk, Lua, and Java (which was presented in an earlier article ). To this regard Java is pretty much like C : all your basic ECS operations (e.g. creating a new Entity, binding a Component to an Entity) are going to be expressed with function calls. The problem is this makes overall code lengthy and tedious, for example to add a Component you would write e.addComponent("whatever", new Whatever()), whereas a more readable way would be e.whatever = new Whatever(). With JS/Lua I looked for a language with metaprogramming, that would allow more control over the syntax. The result is shorter and more readable code, which allowed venturing into more complex UIs. I think having a short syntax is essential to make medium to complex apps. Entitas had it right with their preprocessor. When you sit atop an OOP language, if using a native OOP feature takes less effort than ECS then you might be more tempted to use a few tricks from OOP (storing a lambda directly as Component, allowing methods on Component classes, etc.). I haven't looked at Rust yet but it seems like a fairly good language for ECS.

My personal goal is actually a bit different. Namely to use the ECS GUI architecture with 2D CPU-rendered libraries (I aim for Blend2D specifically - I know the dev for ~12 years already).

Good choice IMHO, I'll definitely be interested in how you develop the Systems doing the rendering.

Yep, this was one of the most enlightening parts! But I feel somehow that all these (G)UI cases can be abstracted into a finite number of such Systems (maybe just few dozen) and thus once this'll be researched, there will be not much need to extend it. I'd call it a "first normal form".

Interesting. I don't have a clue on how to solve this issue yet, please keep experimenting and share your results :)

On the other hand, the core of ECS GUI feels pretty much like an abstract concept fully independent from any implementation. So maybe a detailed architecture standard (imagine CSS, HTML, ...) covering whole ECS GUI could also be the way to go (but such a standard can't be developed without having a working protope beforehand). The research paper (not your thesis) about Polyphony could be also viewed as such a standard 😉.

Agreed. CSS is actually a very good basis to (G)UI Components if you think of it. One would just need to complete the missing bits that aren't style (e.g. saying that an Entity may be hovered/toggled/clicked/dragged). I'd love to see a research effort making a standard out of it.

[dumblob]

Prior to JS, I experimented with ECS on C, Smalltalk, Lua, and Java (which was presented in an earlier article ). To this regard Java is pretty much like C : all your basic ECS operations (e.g. creating a new Entity, binding a Component to an Entity) are going to be expressed with function calls. The problem is this makes overall code lengthy and tedious, for example to add a Component you would write e.addComponent("whatever", new Whatever()), whereas a more readable way would be e.whatever = new Whatever(). With JS/Lua I looked for a language with metaprogramming, that would allow more control over the syntax. The result is shorter and more readable code, which allowed venturing into more complex UIs. I think having a short syntax is essential to make medium to complex apps. Entitas had it right with their preprocessor. When you sit atop an OOP language, if using a native OOP feature takes less effort than ECS then you might be more tempted to use a few tricks from OOP (storing a lambda directly as Component, allowing methods on Component classes, etc.).

Very interesting. Will think about the consequences. Should you have more food for thought, post it here any time!

I haven't looked at Rust yet but it seems like a fairly good language for ECS.

I'm discussing (G)UI with V lang community for almost 2 years now and I think it'd be my preference to try out ECS GUI with V first. On the other hand, just recently I've mentioned Polyphony among the Bevy community (one of the few leading pure Rust 3D engines & frameworks), so Rust definitely seems be another good choice.

My personal goal is actually a bit different. Namely to use the ECS GUI architecture with 2D CPU-rendered libraries (I aim for Blend2D specifically - I know the dev for ~12 years already).

Good choice IMHO, I'll definitely be interested in how you develop the Systems doing the rendering.

Just to clarify - I most probably won't write the code. But I'm preparing requirements for a potential next project involving GUI 😉. I don't even have time to maintain the IMGUI Nuklear.

Interesting. I don't have a clue on how to solve this issue yet, please keep experimenting and share your results :)

Well, there is one rather simplistic idea. Namely to force the run of the whole systems chain over and over if somewhere during the chain a flag was set indicating the need to re-run the chain again (this flag is set to "no" as the very last operation of the chain). This is how it's being traditionally done in synchronous systems (ECS is a purely synchronous system). But yeah, performance might suffer if systems are not optimized for these "shortcut runs". One interesting optimization is that systems can easily check whether the re-run flag is set and return early if they're sure no other systems can depend on their work. This is particularly handy for systems whose main purpose is to have "side effects" - e.g. draw or print something. Coincidentally these systems tend to dominate the time of the whole systems chain. So just this simple optimization could be enough for many (most?) situations.

Agreed. CSS is actually a very good basis to (G)UI Components if you think of it. One would just need to complete the missing bits that aren't style (e.g. saying that an Entity may be hovered/toggled/clicked/dragged). I'd love to see a research effort making a standard out of it.

Me too. Unfortunately I don't know researchers who could do this as pure research, but if I'll stumble upon someone, I'll mention it here 😉.

[traffaillac]

I'm discussing (G)UI with V lang community for almost 2 years now and I think it'd be my preference to try out ECS GUI with V first. On the other hand, just recently I've mentioned Polyphony among the Bevy community (one of the few leading pure Rust 3D engines & frameworks), so Rust definitely seems be another good choice.

Vlang would be great too! Their builtin UI support is a killer feature, I've always been baffled at how "modern" languages make it so hard to build any kind of graphic/interactive stuff. If Vlang had constants with units (e.g. width: 600px) it would just be perfect.

Just to clarify - I most probably won't write the code. But I'm preparing requirements for a potential next project involving GUI 😉. I don't even have time to maintain the IMGUI Nuklear.

Wow didn't know that, imGUI has been mindblowing to me (and still is), I'm sure this project is in good hands ;)

Well, there is one rather simplistic idea. Namely to force the run of the whole systems chain over and over if somewhere during the chain a flag was set indicating the need to re-run the chain again (this flag is set to "no" as the very last operation of the chain). This is how it's being traditionally done in synchronous systems (ECS is a purely synchronous system). But yeah, performance might suffer if systems are not optimized for these "shortcut runs". One interesting optimization is that systems can easily check whether the re-run flag is set and return early if they're sure no other systems can depend on their work. This is particularly handy for systems whose main purpose is to have "side effects" - e.g. draw or print something. Coincidentally these systems tend to dominate the time of the whole systems chain. So just this simple optimization could be enough for many (most?) situations.

At this point we're hypothesizing without actual cases, so this is all conditional to there actually being a need for strong inter-Systems coupling. It might be possible to make Systems declare on which Components they depend (although a bit tedious), or even to detect it automagically. Then you would be able to say before running any System whether it needs to run. Say the information that a System has to rerun is a boolean. Then concatenating the booleans from all Systems would yield a bitset (index of a System is its position in the whole chain). Creating such a bitset is manageable : you make a bitset for each Component that stores whether every System depends on it, then you OR the bitsets of Components that changed. From the bitset of Systems that need to rerun, you iterate on set bits and tadaaa, super fast shortcut runs :) Bonus feature is that you can easily compare the sets of Systems that actually ran at the end of each loop, and thus detect cycles (i.e. System A awakes System B which awakes System A, etc.).

Me too. Unfortunately I don't know researchers who could do this as pure research, but if I'll stumble upon someone, I'll mention it here 😉.

Thanks!