Rethink how the Tres core gets/creates/shares its state

[andretchen0]

The way that the core gets/creates/shares its state has a few bugs that are difficult to address currently. In addition, there are some feature requests that could be productively impacted with a few good decisions related to state.

I'd like to get some ideas and feedback from the community.

(Below take "instance" to mean "an instance of <TresCanvas />".)

Open issues

Shared state

The current version of the core has open issues related to state shared across instances:

• Calling pause immediately after onLoop doesn't do anything #251 – perhaps related
• Multiple instances of useRenderLoop per page share state #252
• Docs extend example throws error #565 – extend shares the catalogue across instances.
• useRenderLoop().onLoop ticks before the renderer can meaningfully render #595

Incomplete context

Sometimes context doesn't have all the expected parts.

• useTresContext().controls.value is null when using Cientos' <OrbitControls /> #418

Feature requests

There are also nice-to-have feature requests that could potentially be solved by inverting dependencies and passing a context to <TresCanvas />.

• Support for Custom Renderers #272
• Use window for pointerevents #581

And from the wishlist

• Physics engine add-on? #450 – marked as closed, possibly upcoming

DX

There's also a DX issue that could be impacted:

• Use default slot props to get Canvas context #572 – in the Vue SFC lifecycle, <script setup> is executed before the <template>. But in a file with <TresCanvas />, <template> is currently where the context is created. So, users who want to use context in <script setup> have to wait until onMounted or use a ref depending on their needs.

Pokes

@alvarosabu @Tinoooo @JaimeTorrealba @userquin

[andretchen0]

Suggestion: NodeOps functions should close over context and a private-to-them store

NodeOps currently doesn't have access to context, so it grabs what it hopes is the correct Scene off of the last passed argument, then shares that across instances with scene.

I think we can make nodeOps easier to write if we:

• have the nodeOps functions close over context
• have the nodeOps functions close over a private store

So an unbound nodeOps function will take the form of e.g.,

insert = (context, store) => (child, parent) => {
    // NOTE: e.g., get the scene
    console.log(context.scene)
    // NOTE: e.g., use the store
    if (store.has(child)) { ... }
}

When the core creates the context, it will call the nodeOps' functions with the context and store, and pass the result to Vue's createRenderer. By doing this, nodeOps functions can simply use context and set/get on their private store.

Issues

• Physics engine add-on? #450
• NodeOps shares scene across multipleTresCanvass #560

Status

Open to discussion.

I'm currently working on a POC that also allows for nodeOps plugins.

[andretchen0]

Investigation: allow a context to be supplied to <TresCanvas />

Can we use dependency inversion and allow <TresCanvas /> to accept an unbound context, i.e., a context that's not yet being used by another <TresCanvas />?

That would allow us to e.g.,

<script setup>
[...]
const context = useUnboundWebGLContext() // return a new, unused, unbound context
const {camera, canvas, extend, useRenderLoop } = context
extend(OrbitControls)
useRenderLoop(...)
</script>
<template>
<TresCanvas :context="context"> <!-- Don't forget to bind the context! -->
  <TresOrbitControls :args="[camera, canvas]" />
</TresCanvas>
</template>

If no context is supplied to <TresCanvas />, then <TresCanvas /> can proceed to create one for itself, using useUnboundWebGLContext.

This would potentially open the door to using different renderers.

<script setup>
[...]
</script>
<template>
<TresCanvas :context="useUnboundCSSContext()" />
</template>

It would also avoid the current need to use a ref or onMounted in order to get the context in <script setup>, which is a DX issue that comes up from time to time.

Issues

• Multiple instances of useRenderLoop per page share state #252
• Support for Custom Renderers #272
• Docs extend example throws error #565
• Use default slot props to get Canvas context #572

Status

Open to comments/ideas.

[userquin]

this looks better

[userquin]

Next steps, add some options to useUnboundWebGLContext:

<script setup>
[...]
// `useUnboundWebGLContext`  will provide the context and so TresCanvas can inject it or create a new one if not injected (maybe will not work but we can get the current instance)
const {camera, canvas/*, extend, useRenderLoop*/ } = useUnboundWebGLContext({
  extens: ['orbit-controls', .....], // <== THIS WAY YOU DON'T NEED TO DO everywhere
  onLoop: (ctx) => { /* some logic */}, <== if callback provided
}) // return a new, unused, unbound context
//const {camera, canvas, extend, useRenderLoop } = context
//extend(OrbitControls)
//useRenderLoop(...)
</script>
<template>
<TresCanvas>
  <TresOrbitControls :args="[camera, canvas]" />
</TresCanvas>
</template>

[andretchen0]

Thanks for the feedback. I definitely think we should allow the user to pass one or more arguments.

useUnboundWebGLContext(p: ContextConstructorParameters) (or whatever)

In an effort to invert all the deps, maybe we just let the user bring their own e.g., renderer if they're not happy with the default?

Same for onLoop. We could let the user bring their own update/render loop. The Tres core will just tick whatever function is passed. If that's a simple onLoop function that can pause rendering, then that's what they get. If it's an entire game-framework-like update/render loop, then that's what they get.

Maybe I'm straying too far into "configuration hell" though. I think lots of our users just want to get stuff on screen with the defaults as simply as possible.

extends

I like the idea of allowing the user to extend the library using the argument. Maybe we turn extends into a constructor to keep the mental model above consistent. (It doesn't currently have a constructor and all the contexts share extends, iirc. I believe that's behind a few bug reports. It'll get worse if we eventually have plugins that need to extend.)

To keep the same mental model as the above constructor args, maybe the constructor looks like: new Catalogue(extensions:{}) and you pass {catalogue: new Catalogue({OrbitControls})} ?

onLoop

Just as an aside, at this point onLoop/useRenderLoop is a subject of discussion in its own right. What's certain (to me?) is that each context needs its own way to tick. But useRenderLoop is shared across instances atm.

What's sorta certain to me is that useRenderLoop is mostly not the abstraction that I want. I usually want to pause or slow down updates not renders.

And I think @JaimeTorrealba would like <MyComponent :on-loop="({delta}) => doSomething(delta)" />.

All that to say, there's a lot going on with onLoop!

[andretchen0]

An idea for hiding unbound context creation

Discussion context

Continued from: #624 (comment)

@andretchen0 do you think we could do it without a breaking change or it would be better to delay the release and add this to v4?

Personally, I had been holding off on the points in #578 – how to get a new context/render loop before <TresCanvas /> exists, plugins – until after v4, as I didn't want to complicate things that were already underway.

For exploration

In order not to break user code, to tie a context to useRenderLoop, we'd have to pretend that:

<script>
import { useRenderLoop } from '@tresjs/core'
useRenderLoop().onLoop(...)
</script>

<template>
    <TresCanvas />
</template>

... is the same as something like ...

<script>
import { useUnboundTresContext } from '@tresjs/core'
const context = useUnboundTresContext()
context.useRenderLoop().onLoop(...)
</script>
<template>
  <TresCanvas :context="context" />
</template>

That's probably possible, at least to an extent.

We could create a global unbound context behind the scenes if we're not in a subtree that provides a context – i.e. we're not "inside" a <TresCanvas />.

Then <TresCanvas /> could check if there's an unbound context when it's created:

• if there is an unbound context, use it
• if there's not, create a new one

Hiding this from users means they don't have to explicitly create an unbound context, which is nice. "It just works." At least in cases most of our users seem to produce: using a single <TresCanvas />.

But it'll "just not work", and least in cases where scriptA and scriptB begin execution before TresCanvasA is created. I don't know enough about lifecycles to know if that's an issue.

Compromise?

I suppose as a compromise, we could offer an "easy" API for most use cases – useUnboundContext is hidden behind useRenderLoop. And an "advanced" API where users can call useUnboundTresContext and get a handle on an unbound context.

[alvarosabu]

Hey @andretchen0 I just addressed some of your concerns regarding the onLoop/useRenderLoop here #252 (comment) but to give context to @userquin and @JaimeTorrealba

Multiple loops on the scene graph

Create a separate update loop that is distinct per user component

@andretchen0 The thing is, ThreeJS renders all the scene graph at once when using renderer.render(scene, camera) and that method is called with onLoop every frame. To achieve the level of granularity where you can pause individual components from rendering, we would need to create an onLoop instance per object, attach it to their localstate and then overwrite the onBeforeRender method of the THREE.Object class to conditionally render based on the paused state https://threejs.org/docs/#api/en/core/Object3D.onBeforeRender

This is certainly something I'm not eager to do because it adds a level of complexity and possible side effects that would make it difficult to maintain, but to consider it I would like to understand how the end user would benefit from having individual loops and if there are not end-user alternatives, for example

Imagine a scene with 100 objects, all of them with event listeners to handle.

Most uses I've come across of useRenderLoop are for animating attributes. In that context, calling pause() and then not rerendering, e.g., when the screen is resized is counter to my intuition.

Wouldn't be easier to just not render the object with a v-if instead of using the individual onLoop to stop the rendering?

isPaused = ref(false)

<Box v-if="isPaused" />

Let's continue the dicussion

[andretchen0]

Hey @alvarosabu !

Imagine a scene with 100 of objects, all of them with event listeners to handle.

Sure thing. Maybe it's more complicated for the Vue/Three/Tres combo than I'm currently aware.

Like rendering, updating would use graph/tree traversal. If we separate concerns properly:

• The traversal code doesn't know anything about the individual onUpdate listeners, except how to access them and how to call them.
• The onUpdate listeners themselves are pushed down to the updating components. They largely don't need to know about tree traversal; they just know that their onUpdate will be called.

Wouldn't be easier to just not render the object with a v-if instead of using the individual onLoop to stop the rendering?

The v-if would insert/remove the component from the graph, which serves a different purpose. Here, I'm just talking about updating (or not) a component/subgraph in the graph.

In the cases I see – mostly looking at @JaimeTorrealba 's code, fwiw, because we review each other for Cientos – useRenderLoop().onLoop() mostly isn't used to render stuff. It's used to "step" or "tick" a variable or attribute. Pausing a "tick" on a variable usually shouldn't affect rendering or remove the component, at least to my mind.

Fwiw, it's pretty common for game frameworks to have separate loop for update and render.

And running a render loop and multiple update loops at various speeds isn't uncommon either. E.g.,

• the physics engine code might be simpler to write at a fixed framerate – that lets coders drop delta or dt
• the rest of the gameplay might update at a variable framerate – with a goal of 30 fps
• and finally the renderer might run at 60 fps

IIRC, this is how the game Dead Cells updates/renders.

Physics simulations are in the scope of the @tresjs/rapier package, but the speed modifier couldn't be just a ref on the end user component?

That's a possibility, but it requires deltaModifier to be pushed down into the implementation of the listeners.

Imagine you have this tree:

<Game>
<Hud>
    <Health>
       <Bar />
       <Number />
    </ Health>
    <Energy>
       <Bar />
       <Number />
    </ Energy>
</Hud>
<GamePlay>
    <Level>
      <Hero />
      <NPC />
      <PowerUp />
    </Level>
</GamePlay>
</Game>

If you want to slow down the Hud updates due to some effect – e.g., the player has been hit with an ice sword, lol – with the deltaModifier built into the update loop, Hud could look like this:

<Hud :deltaMod="frozen ? 0.25 : 1.0" >
    <Health>
       <Bar />
       <Number />
    </ Health>
    <Energy>
       <Bar />
       <Number />
    </ Energy>
</Hud>

None of the child components' listeners need to be rewritten. The delta passed to their listeners will be the parent delta * deltaMod. And each component can modify deltaMod for their own subtree just by changing their own deltaMod.

If we do the above with a ref, we've got to change the implementation of several components.

[alvarosabu]

Hi @andretchen0 thanks for the clarification, I think I understand better now what would be the use-case for updating instances.

Additionally, it might be beneficial to touch upon how Three.js itself manages update and render loops, to provide further context. As TresJS serves as a custom renderer for Three.js within the Vue ecosystem. Three.js typically utilizes a single render loop, managed via requestAnimationFrame, where updates to object properties or animations need to be manually synchronized with each render call. This design choice encourages a clear separation of concerns between updating scene state (like object positions, animations, etc.) and the actual rendering of the frame.

To be fair useRenderLoop is available to offer the user (mantainers, package authors, end users) a way of syncing updates (animations etc) with the rendering rate. Providing the time between renderings (delta) independent of the frame rate of the device.

But we don't force it's use, is just convenient as is a similar DX as ThreeJS and others custom rendered offer. But cientos or any abstractions could easily implement its own step of tick like this:

let lastTime = 0;
let accumulatedTime = 0;
const fixedUpdateTime = 1 / 60; // Fixed update time step (e.g., 60 times per second)
let speedModifier = 1.0; // Normal speed

function animate(now) {
    requestAnimationFrame(animate);
    const deltaTime = (now - lastTime) / 1000; // Convert to seconds
    lastTime = now;

    // Accumulate time and update at fixed intervals
    accumulatedTime += deltaTime * speedModifier;
    while (accumulatedTime >= fixedUpdateTime) {
        update(fixedUpdateTime); // Update with fixed time step
        accumulatedTime -= fixedUpdateTime;
    }

    updateObject(); // Update object
}

I even think we do that on the devtools to render the fps and memory graphs. What I want to clarify is that is not possible to define a rendering loop per object, an updating loop would be another story.

Fwiw, it's pretty common for game frameworks to have separate loop for update and render.

Just to clarify tho, our primary scope is not to be a game framework/engine, we are a ThreeJS custom renderer for Vue. I'm pretty sure there would be more optimized options than TresJS to build games although I love seing games created with it.

But since you mention the usecase of GUI and HUD, considering you are using Vue, wouldn't be more appealing to create the graphic UI interfaces with Vue and leave the gameplay to Tres? I mean, why creating a heart bar at the top with ThreeJS if you can easily do it with Vue?

<Game>
// Vue UI
<Hud>
    <Health>
       <Bar />
       <Number />
    </ Health>
    <Energy>
       <Bar />
       <Number />
    </ Energy>
</Hud>
// Tres renderer
<GamePlay aka TresCanvas>
    <Level>
      <Hero />
      <NPC />
      <PowerUp />
    </Level>
</GamePlay>
</Game>

Look at @Dekier great example: https://marcin-dekier.com/, the items HUD is just a vue component sharing a state with Tres.

[JaimeTorrealba]

I don't know if this help, but time ago a read https://gameprogrammingpatterns.com/ (recommended book). There's a special chapter about game loop and update() functions. As we don't own the loop (which its own by the browser the RAF API) getting mess with it could lead to bad practice and side effects that are hard to debug or understand. (they in the book talk in detail about this)

I'm more in the @alvarosabu side we should not do a renderloop per object... if the users need to pause (or slow) some specific component for now I would say, it has to be done manually. Also, I don't believe we're a game engine, or we should act like one.

More importantly for me is that, we don't have enough hands... I believe there are Other priorities to Tres (like other package) maybe in the future we could come back to think this issue better, or create a game engine utils pkg.

[andretchen0]

@alvarosabu @JaimeTorrealba

Thanks for the discussion. I hear you.

It sounds like the way forward is to focus on the open issue: figure out how best to pause/resume useRenderLoop so that it only affects a single instance.

Any ideas here?

We could provide a useRenderLoop to descendants of a TresCanvas. This might look like how /src/composables/useTresContextProvider provides a contexts to descendants.

But that means useRenderLoop isn't available until after TresCanvas's setup runs. And users don't like that.

---

@JaimeTorrealba

Nice book recommendation! ⭐️⭐️⭐️⭐️⭐️