How to use a stream of images to be the background of SceneView?

[sandeepbangalore]

I have a phone with ARCore enabled and I am sending the video feed as images to another phone. I want to use this stream of images as a background in the other phone instead of invoking the Camera. ExternalTexture seems to be the way to go but I am having a problem understanding how to implement it. Though there is a sample project which renders a video, I would like to understand how I can use the same concept towards real time video rendering. I am using OpenTok for sending and receiving the video feed. Some help in this direction is greatly appreciated.

[dsternfeld7]

Does the information in issue 274 that shows how to render an ExternalTexture as a background help you with this?

[sandeepbangalore]

I did have a look at issue 274 but it seems like those images are being passed through the camera's setPreviewDisplay. Is there a way to do it without having to use the camera API?

[dsternfeld7]

Yes, you can use the technique described in issue 274 to render an ExternalTexture as a background, and then render anything you want to the ExternalTexture's surface.

[sandeepbangalore]

I will try that out, thanks. Will get back to you if I have any issues.

[sandeepbangalore]

I am able to get a stream running on the background but the background is present only on the top half of the screen (As shown in the image below). Is there anything that I am missing?

OBJ File

v -1.000000 0.000000 0.000000
v 1.000000 0.000000 0.000000
v -1.000000 1.000000 0.000000
v 1.000000 1.000000 0.000000
vt 1.0000 0.0000
vt 1.0000 1.0000
vt 0.0000 1.0000
vt 0.0000 0.0000
vn 0.0000 0.0000 1.0000
s off
f 1/4/1 2/1/1 4/2/1 3/3/1

Custom Material File

material {
    "name" : "Camera Background",
    "defines" : [
        "baseColor"
    ],
    "parameters" : [
        {
           "type" : "samplerExternal",
           "name" : "cameraTexture"
        }
    ],
    "requires" : [
        "position",
        "uv0"
    ],
    "shadingModel" : "unlit",
    "vertexDomain" : "device",
    "depthWrite" : false,
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);

        vec2 uv = getUV0();
        material.baseColor = texture(materialParams_cameraTexture, uv);
        material.baseColor.rgb = inverseTonemapSRGB(material.baseColor.rgb);
    }
}

[dsternfeld7]

The y coordinates in your obj file are going from 0 - 1, but they should be -1 to 1.

[sandeepbangalore]

I did try it with -1 to 1 but had no luck. I anyways found a work around to my bigger problem and have decided to go with an ArSceneView itself. Thanks!