integrating the aframe multi camera with 3DStreet

[srothst1]

Hi @diarmidmackenzie!

I have had a lot of fun experimenting with your aframe-multi-camera tool. I'm currently working to integrate this tool into the next version of our 3DStreet viewer. I am working out of the new-coat-of-paint-2.0 branch in the 3DStreet repository:

• https://github.com/3DStreet/3dstreet/tree/new-coat-of-paint-2.0

Here are a few important parts of my implementation so far:

I am importing your code via. CDN in the head of my HTML file.

<script src="https://cdn.jsdelivr.net/gh/diarmidmackenzie/aframe-multi-camera@latest/aframe/cursor.min.js"></script>
<script src="https://cdn.jsdelivr.net/gh/diarmidmackenzie/aframe-multi-camera@latest/src/multi-camera.min.js"></script>
<script src="https://cdn.jsdelivr.net/gh/diarmidmackenzie/aframe-multi-camera@latest/src/viewpoint-selector.min.js"></script>
<script src="https://cdn.jsdelivr.net/gh/diarmidmackenzie/aframe-multi-camera@latest/src/mirror.min.js"></script>	

I am also adding divs with ids viewport1, viewport2, etc. to a toolbar that is located in the top part of the page.

<div class="menu-img" id="viewport1" style="border-style: solid; width: 200px; height:130px">
</div>

Finally, I am adding a set of second cameras to my a-scene

<a-entity id="camera1" secondary-camera="outputElement:#viewport1" position="-8 1.6 0" rotation="0 -90 0"></a-entity>
<a-entity id="camera2" secondary-camera="outputElement:#viewport2" position="-8 1.6 0" rotation="0 -90 0"></a-entity>
<a-entity id="camera3" secondary-camera="outputElement:#viewport3" position="-8 1.6 0" rotation="0 -90 0"></a-entity>
<a-entity id="camera4" secondary-camera="outputElement:#viewport4" position="-8 1.6 0" rotation="0 -90 0"></a-entity>

While I think that I am heading in the right direction, I am getting an error that crashes my application

three.js:20378 Uncaught RangeError: Maximum call stack size exceeded
    at setProgram (three.js:20378:1)
    at WebGLRenderer.renderBufferDirect (three.js:19887:1)
    at renderObject (three.js:20278:1)
    at renderObjects (three.js:20254:1)
    at renderScene (three.js:20205:1)
    at WebGLRenderer.render (three.js:20084:1)
    at o.render (multi-camera.js:66:27)
    at PMREMGenerator._textureToCubeUV (three.js:11811:1)
    at PMREMGenerator._fromTexture (three.js:11680:1)
    at PMREMGenerator.fromCubemap (three.js:11619:1)

Any suggestions on how to resolve this issue?

cc @kfarr for context

[kfarr]

The specific error appears to be related to the cubemap for the sky; for short-term testing you could try removing the sky cubemap texture and see if that works

[diarmidmackenzie]

It's not obvious to me what the problem his here. Following @kfarr's suggestion about the cubemap, I tried adding that to one of the examples, and it seems to work OK without errors:
https://diarmidmackenzie.github.io/aframe-multi-camera/tests/multi-screen-with-cursor-plus-cube-env-map.html

For multi-camera to work, we override the A-Frame render() call. From the stack trace, my guess is that the A-Frame render call is being used to prepare CubeMap textures in some way, that's invoking the multi-camera override verson of render() and somehow that's causing recursion, leading to the "Maximum call stack exceeded" error.

But from my test it looks like something more than just adding a cubemap is needed to create this problem, and I'm not sure yet what. I'm going to try cloning your 3dstreet repo and see if I can see what's going on there by running that locally.

As an aside, if you are't using the specific functions (and I don't think you are), there's no need to include the viewpoint-selector.js and mirror.js files in your HTML.

[diarmidmackenzie]

OK, I'm not sure how to reproduce the problem with 3dstreet.

I cloned the repo, switched to the new-coat-of-paint2.0 branch, and viewed index.html.

The multi-camera function seems to work.

I checked the console logs and there are a few errors, but not the specific error you mentioned above (and the errors that do exist don't look to be related to multi-camera).

Can you help me understand how you reproduce this?

Based on the stack trace, I have some idea what might be going on (I'll add another comment), but it will be hard to progres without a way to reproduce the problem.

[diarmidmackenzie]

In terms of what is going on (based on the stack trace).

• something in your scene is triggering Three.js to generate a Prefiltered, Mipmapped Radiance Environment Map (PMREM) from a cubeMap environment texture: https://threejs.org/docs/#api/en/extras/PMREMGenerator.fromCubemap
• to do that Three.js uses the WebGL Renderer - but with a bespoke scene & camera (not the main scene & main camera).
• the multi-camera component overrides the render function on the renderer. The way you have set things up, it invokes a series of additional renders within the render loop, to render the overlay images.
• unfortunately (this is the bug), it does that indiscriminately, so this is happening whenever the renderer is used, even if it's being used for a completely different task (e.g. generating the PMREM).
• by this point, all bets are off as to what happens next. My guess is we end up with recursion because in the process of trying to render one of the additional camera views, Three.js notices we don't yet have the PMREM we need, and so decides to render that, leading to an infinite loop :-(

There's probably a fairly simple fix.

In the add-render-call system, we should check the scene and camera, and only invoke the configured additional render calls if the scene and camera values match the values we expect (a little care needed to define "the values we expect" appropriately for all the use cases of multi-camera, but I'm sure that's a tractable problem).

I'm happy to try to code a fix for this, but it would be extremely useful to have a repro scenario to test out a fix with. Do you have any idea what in 3dstreet is triggering the generation of the PMREM, and why that's not happening in the latest code on the new-coat-of-paint2.0 branch?

[diarmidmackenzie]

Based on the above understanding, the following fix should hopefully work:
https://github.com/diarmidmackenzie/aframe-multi-camera/tree/issue2-fix-PMREM-generation-issue

("the values we expect" turned out to be simple. aframe-multi-camera doesn't use recursion in its rendering - the sub-renders always directly invoke the original WebGLRenderer render function. "Infinite tunnel" effects aren't generated through recursion, but in fact build up frame over frame. Net of that is that the add-render-call system should only act when rendering the main scene with the main camera).

[diarmidmackenzie]

@srothst1 if you can confirm the fix worked for you I'll merge it into main.

Or if it doesn't work, I'm happy to take another look.

[diarmidmackenzie]

I merged this fix, so I believe this should be solved now.