Initial thoughts on the candidates

[Frooxius]

I'm just compiling some initial thoughts on the candidates right now.

This is not a final decision and things can change as more research is done.

I'm just collecting thoughts and starting discussion on each one. If you'd like to contribute to discussion, comment on the appropriate thread!

[Frooxius]

Bevy

This one seems quite promising and modern, but also very immature, so waiting for feature parity might take a long time.

Right now it doesn't even seem to support VR rendering for example, so if we were to consider using this one, we'd have to likely wait a good while.

I think because of that it's not likely one we'd pick.

[MalekiRe]

i use bevy in production at work every day, but I am not happy with the quality of the xr integration I helped create, its not up to my current standards and I would like to rewrite it from scratch. Its served me well the past year writing dozens of different virtual reality apps, running on Samsung galaxy xr, Varjo, Quest, Vive Focus, etc. Using things like hand tracking, controller rendering provided models from openxr, camera access, and more.

[MalekiRe]

Its also important to note that bevy's rendering road map very much includes modularizing the renderer and making it easy to swap out different pieces, and its already partway there, separating the concepts of meshes, cameras, lights, and other things from the renderer itself. This is a big point in favor IMO for a very long term facing project like yours, as it will mean over the next few years you will be able to specialize it as needed

[Frooxius]

Hmm that'd be good too.

My main concern is how long it would take for us to be able to use it as a replacement though. Does that mean waiting several years for its development to progress before we can being our own work?

Or is this going to happen much sooner?

[MalekiRe]

I have no immediate plans, but we're always open to contributions ( or forks ), the nice thing about bevy is that its really modular, so its easy to polyfill functionality. Its hard for me to judge if its ready for your use-case, there may be bugs or features you need that aren't there yet and I don't know how willing your team is to work on contributing or building those, but from my perspective I would say its ready enough to be used and relied on, its relatively battle tested.

[MalekiRe]

I think an in-depth look at the API surfaces exposed, and examples provided, by the different projects, and maybe even quickly spinning up a subset of resonite features ( just meshes and pbr shader for example from resonite to the renderer in VR ) in all the projects listed in this github discussion wouldn't be out of the question?

[Frooxius]

Godot

Potentially best candidate out of the "bigger" engines with large community support. The backing of community and even bigger companies is a really good plus.

Main concern would be separating out the renderer itself and reducing overhead. The feature parity seems close enough that would likely be doable.

The biggest plus is that it's a project that will likely keep getting a lot of support and work far into the future.

[CometKuma]

How did I not see VRAM mentioned? I must be more tired than I thought lolol

But yeah... That's a big issue unless the renderer is easy to modify. would probably depend on how well documented the renderer is. good luck team!

[Dusty-Sprinkles]

I don't have the technical skill to really investigate it myself but most of the candidates should probably be evaluated for small but large unity-isms like this, losing out on a ton of vram space from a simple design decision is probably more likely than not

[Merith-TK]

Honestly I would love a godot based renderer since that already gets the foot in the door for dedicated mobile clients such as for the quest 3, or for mobile android,
which yes would run into performance limitations (likely why the neos-legacy mobile friendly toggle is in the world settings still)

[Frooxius]

@Merith-TK I don't think Godot is the only one that can achieve that though. A number of the other candidates also seem to be potentially viable for this.

[Merith-TK]

The main idea behind mine is godot already is a familiar platform (and community work has already been done in this direction) that it is a VIABLE path to consider for the future.

Ultimately performance benifits and ease of integration would outweigh such community work regardless, so it is ultimately up to what would be best suited.

[Frooxius]

Renderide

Probably a top contender for a candidate given it's been designed for feature parity and can run majority of Resonite content with it looking nearly the same.

This would speed up the phase 3 and 4 considerably and gives a lot of confidence in maintaining our principles of long term content compatibility.

My main concerns are currently following:

Performance

From what I've seen this is generally lower than Unity based renderer at the moment, so I'd need to understand how this can be improved to at least match the Unity one.

Long term maintenance/support

Since this is a custom project, another worry I have is that it would end up abandoned and we'd be relatively stuck with it. However it being open source and community making contributions towards it too alleviates a lot of it.

However it's still unlikely to get a lot of movement like the bigger projects do from their broader communities, requiring more involvement from our end. It's not a deal breaker on itself - but would be if it meant that other concerns (like performance) don't get addressed.

If we go with this candidate, I'd like us to potentially onboard DoubleStyx to collaborate more closely

AI usage

Generative AI was used at some point to assist in development. We don't use generative AI to code Resonite, it's designed and coded by us.

However from my understanding once this concern was raised, the code was rewritten by hand, which alleviates that concern, but I'm still worried some people can be uncomfortable - though most of the projects allow some uses, so if we wanted to be 100 % purist about this, we wouldn't have any actual options left.

Us adopting this project officially would also let us enforce our principles going forward (or well for our fork of it).

[FoxoBread]

Performance for me tended to be on par, or higher than the Unity renderer, which I found interesting.
I suspect that is due to the Unity renderer being ran through Proton, or Renderide utilising my compute resources better, per AMD's more refined software end on Linux.

[Comptr-user]

@FoxoBread
tl;dr: Unity performance issues probably aren't due to running under Proton, whether or not Renderide has better performance than unity is "it depends," and sorry for the wall of text. I just wanted to be really thorough and cover as much of what I know.

Proton is a translation layer that takes windows commands and translates them into Linux ones, and it's designed to have minimal overhead.
DXVK--the library Proton uses to translate DX11 calls into Vulkan ones--is really good at its job to the point where some games, like Team Fortress 2, can see a slight performance boost with it on windows. There are dozens of videos on youtube comparing performance between running a game through proton and running it natively on windows with identical hardware. A large amount of games match windows' performance, and some can even outperform it; I really don't think Proton is the culprit for performance problems, and it's probably instead just Unity being Unity™.

Everything below this line of text isn't really related to your comment, Foxo, but I wanted to include it anyways because I felt it was important

In my testing with desktop mode, Renderide's performance is a massive "It depends" with it usually being worse than Unity.

There are some scenarios where Renderide completely outclasses unity in performance, like viewing 160 heavily overlapping spotlights in a grid world. Doing that nukes Unity's performance resulting in a maximum of ~20-30fps regardless as to what I have my hardware set to in LACT or cpupower. Renderide meanwhile manages to reach ~100fps in the same situation even if my GPU is set to my power-saving profile.

On the other hand, there are some situations where Renderide completely falls apart in performance while Unity is at least survivable. I made an item a while ago with the purpose of trying to get a better idea how unity handled instancing. It is 2^16 identical triangle meshes arranged in a rough grid shape. The short of it is Unity is weird and evil and hates me specifically, and only instances them in batches of 511, resulting in a stuttery ~30fps even at maximum CPU/GPU clocks. Due to frustum culling performance recovers if you look away from the triangles. Meanwhile, Renderide rarely reaches double digit FPS in this setup, averaging around 8fps while only going up to ~12fps when I'm not looking at the triangles.

Sidenote: Some testing I did to check the accuracy for stuff I cut from this comment have lead me to believe that something funky (derogatory) is going on with Renderide's Frustum Culling. The 2^16 triangles tank performance when you aren't looking at them even though they should be culled away, and I was seeing similar issues with the USFN Avatar station. There's a large structure that gets deactivated while you're in spawn due to how you can't see it at all, but it's otherwise always active. In unity it can easily be frustum culled when you're not looking at it, but for some reason it existing always causes a slight performance drop in Renderide even when it should be frustum culled. Might see if I can do something about that, but I doubt I'll be able to make any useful changes considering my whole "little to no programming experience" thing.

[Frooxius]

@FoxoBread @Comptr-user It might be worth adding any benchmarks and tests to the requirements document. The more info we have, the easier it'll be to make an informed decision.

[Dusty-Sprinkles]

if you do fork it Please rename it the render-ide/ite is so confusing lol

[Xlinka]

i do still contribute to this project... just on a seperate branch as im not touching main

[Frooxius]

Stride

This engine being in C# is probably the biggest strength of it, which could integrate better.

Smaller compared to Godot, so probably less support, but still some. .NET Foundation is nice in this regard.

Feature parity seems kinda... meh? Like would need a fair bit of extra work to bring it up.

I don't think the C# part is enough to prefer this over Godot, which seems more mature and has much more weight behind it, so probably unlikely for us to pick this one.

[Comptr-user]

I've heard that it's apparently pretty easy to strip out the game-engine stuff from stride and just use it for rendering, however I don't remember where I heard this or if it was a reputable claim.
If this is the case, then that could be a plus for stride, as we could avoid some of the overhead of "using a game engine just for its renderer" through simply removing most of the game engine itself.

[Frooxius]

Ogre Next

I understand this candidate the least I think.

Seems like limited VR support. Smaller community. Feature parity okayish, but would need more work.

I don't think it's interesting choice over Godot / Renderite in particular, I don't see much that would highlight this.

[polyjitter]

I think the most interesting about it in comparison to Godot, really, is that it's purely a rendering engine, so you don't really have to worry about stripping out or building out the other bits. With Renderide in the mix, however, I'm not sure that's that much of a concern.

[Frooxius]

@polyjitter That's a good one yeah! That would definitely make it easier to adopt like that without a lot of overhead.

[TanadrineStudios]

I have no experience with Ogre Next, but I definitely understand why they needed to start over from original Ogre which I have used. It is extremely over-engineered and messy in some areas of the implementation and it simply cannot be adapted to fit modern standards.

[Dusty-Sprinkles]

oh also you might want to fix the Renderite -> Renderide on this too

[Frooxius]

The Forge

Probably more interesting than Orge, but also kinda mix. A lot of feature parity is missing and would need fair bit of work to bring it up there.

The lack of documentation is probably biggest concern.

Other one is that we'd have to write a lot of our own systems - which makes sense this being a framework, but also quite a big commitment and we'd need to maintain our own rendering engine going forward.

Because of that, I don't think there's much advantage over Renderide, which already does that, but uses more modern underlying frameworks and Rust based instead of C++, which is safer.

[Earthmark]

This is a higher-order question, I can delete this comment afterwards.

One of the pain points I've had with trying at another renderer has been the sheer volume of messages. It's not hard, but it's a lot of fields and types. It feels like many of these renderers would be easy to implement prototype wise, except for IPC and message formatting.

Froox, how would you feel if the renderite messages shifted to be schema-first instead of defined in C# first and foremost?

Define a json or yaml document of the messages and field types, and have that be the authority for Renderite and any other language. If new fields or messages are added, the dependent codegens can regenerate as needed.

Could also be generating a schema off the C# side, but that feels possibly messy (say a command reorders the Write calls in the Pack method vs the field order).

There's some out of the box generators like cptnproto and flatbuffers, but most have some rather nasty asides (flatbuffers in C# writes memory backwards for instance). There may be a schema-first one that's out there, but I figure seeing if a non-C# authority is viable is first anyways.

[Frooxius]

@Earthmark We'd probably shift for something that we can easily generate multiple bindings for, but that will depend on which renderer we end up going with.

We would very unlikely end up using something like flatbuffers, the IPC protocol is heavily optimized for maximum performance and generally keeping the memory layout.

This is something that we'd likely do in phase 4 though, rather than at this stage - for selection of the candidate it doesn't matter that much.

[Dusty-Sprinkles]

@Earthmark We'd probably shift for something that we can easily generate multiple bindings for, but that will depend on which renderer we end up going with.

We would very unlikely end up using something like flatbuffers, the IPC protocol is heavily optimized for maximum performance and generally keeping the memory layout.

This is something that we'd likely do in phase 4 though, rather than at this stage - for selection of the candidate it doesn't matter that much.

Wouldn't it make sense to streamline the bindings earlier, since its going to be hard to get ground truth benchmarks without actually being able to test the renderers that haven't had fleshed out candidates created yet?

Unless you're planning on creating integrations for all of them?

[Frooxius]

@Dusty-Sprinkles I'm not expecting actual implementations in this phase.

That would require creating functional implementations for all candidate renderers, which is a huge amount of work that we don't have bandwidth to undertake (and wouldn't be good use of our time) - because now instead of just making one renderer implementation, we're making like 5-6.

For benchmarks, you'd more likely setup similar scenes in different engines and see how it performs with different scenarios.

[Earthmark]

My post was more if you would consider it an affront Froox, the library is open source and template generators are pretty easy to make, it seems community-contribueable from my perspective.

I was more asking if one was made, and it generated binding classes for C# that matched the implementations currently in this library, had feature parity, and were to your standards; would you accept them as part of the library? (Non binding of course)

I'm not saying it's a blocking thing either, just if the work was put in by community members, would that be an acceptable path. If it doesn't happen, no harm done, go on as your plans dictate. Even if a schema approach is merged, maybe it gets dropped after if the chosen library is C# based.

If a schema was made, community members could more easily make prototypes themselves, regardless of what renderer is chosen by YDMS. I figure no matter what YDMS picks, there's no reason another renderer couldn't exist like it does today with Renderi(de|te?). No reason a new renderer could be picked again down the line anyways.

The main issues I've seen with making renderers that are built on top of existing engines is largely one of documentation and error prone message boilerplate, both of those things can be addressed in this repo. The IPC and command sequencing can be documented with markdown files, the learning Doublestyx, Cyro, and others had to do can be more easily passed on. With schema defined messages, the copy-paste brute force can be simplified and be far less error prone.

PS: My mentioning flatbuffers and the like was more to say that they exist, but are not as viable. I was proposing using the existing IPC mechanisms and such. My entire question was more about if the structure that defines the messages gets another layer on top, I was not proposing a change to how the layer works at runtime.