How to display video frames with deck.gl ?

[jrabary]

I'm new to deck.gl and wondering how to display video frames with it.
I would like to build an app like the ascii demo but instead draw the actual frames.
Is the BitmapLayer suitable for this task or do I need to write a custom layers ?
Any hints will be helpful.
Regards

[Pessimistress]

The BitmapLayer does not support video right now. I think this is a reasonable use case and we should consider supporting it. @xintongxia

I put together an example here: https://codepen.io/Pessimistress/pen/pmmbRj

[jrabary]

@Pessimistress Thanks for the example. Is there something missing it the code ? The image in the canvas is not updated when the video is playing.

[Pessimistress]

Are you using Safari? WebGL1 does not have good support for non-power-of-two texture sizes (and the demo video demensions are not power of two). You will need to resize your source video.

[tsherif]

To get an non-power-of-two texture to work in WebGL 1, just change the wrap mode to CLAMP_TO_EDGE (and turn off mipmaps, which was already done here), i.e. in the codepen example, change the texture construction to:

this.setState({
  bitmapTexture: new luma.Texture2D(gl, {
    width: 1,
    height: 1,
    parameters: {
      [gl.TEXTURE_MIN_FILTER]: gl.NEAREST,
      [gl.TEXTURE_MAG_FILTER]: gl.NEAREST,
      
      // THIS IS THE NEW PART
      [gl.TEXTURE_WRAP_S]: gl.CLAMP_TO_EDGE,
      [gl.TEXTURE_WRAP_T]: gl.CLAMP_TO_EDGE
    },
    mipmaps: false          
  })
});

[Pessimistress]

Thanks @tsherif ! Codepen is updated.

[jrabary]

Thank you. It's working great!

[jrabary]

@Pessimistress Just to understand what's going on. Does the fact that we set _animate = true forces deckGL to call layer.draw every timestep ?

[Pessimistress]

That is correct. If you want more control over this, you can manually trigger redraw:

let animation = null;

const videoBitmapLayer = new VideoBitmapLayer({...});
new DeckGL({...});

function animate() {
  videoBitmapLayer.setNeedsRedraw(true);
  animation = requestAnimationFrame(animate);
};

video.onplay = () => animate();
video.onpause = () => cancelAnimationFrame(animation);