ExoPlayer Custom Renderer

[dhruvsingla273]

Hi All,

I am trying to create a player in which I want to do processing on frames of the video.
So, I need to get each frame of the video, do some processing on it, and have to then render it to the Screen.
I searched but couldn't find a compelling method that can be fast enough to render.

Can someone guide me for this how to make the custom renderer.

[droid-girl]

Hi @dhruvsingla273,
You can use .setVideoEffects in ExoPlayer to achieve this:

1. Create a custom effect by extending BaseGlShaderProgram. You can find a few examples in the effects module, for example, ColorLutShaderProgram
2. Do frame modifications in drawFrame method in your custom shader program
3. in ExoPlayer, use setVideoEffects and set your custom effect.

I think this should help you to achieve your goal.

[dhruvsingla273]

Hi @droid-girl ,
Thanks for the reply.
Sorry for not being clear about the processing part in the last question.
Actually I need to apply a TFlite model to each of the frames and I think shaders can't do that.
I tried using MediaCodec to custom extract the frames and then apply the model and render them (https://github.com/duckyngo/Fast-Video-Frame-Extraction).
It worked but I need to do it within ExoPlayer.

I tried making a custom Renderers Factory from MediaCodecVideoRenderer, but the ByteBuffer I got was empty as I got to know that Media Codec runs in Surface mode and hence it is not storing anything in buffers to improve codec performance. I could try to set MediaCodec to byteBuffer mode to get the byteBuffer but I read it is very slow. (Also can you tell how to set this mode change for MediaCodec in ExoPlayer)

Could you suggest a different approach?

[dhruvsingla273]

I have one more question.
If I can get a steady stream of a Bitmaps or ByteBuffer with my final frame data,
can I somehow use ExoPlayer to render them using the player?
Means I won't be having all the frames at a single time, I can have like say 2 sec of frames and then continue processing other frames and keep passing to ExoPlayer to render.

I know it might be confusing, please ask any clarification on the question you may need.
Thanks again

[dhruvsingla273]

Hi @droid-girl
Have you found any way to integrate TFlite model?

[droid-girl]

Hi @dhruvsingla273 ,
Let me first provide a short summary on the approach you can take to integrate with TFLite:

1. Extend [BaseGlShaderProgram] as recommended in the earlier comment (https://github.com/androidx/media/blob/release/libraries/effect/src/main/java/androidx/media3/effect/BaseGlShaderProgram.java)
2. in drawFrame method you want to read frame to be able to pass it through TFLite:

    ByteBuffer pixelBuffer = ByteBuffer.allocateDirect(width * height * 4);
    Bitmap bitmap;
    int texId;
    try {
      int[] boundFramebuffer = new int[1];
      GLES20.glGetIntegerv(GLES20.GL_FRAMEBUFFER_BINDING, boundFramebuffer, /* offset= */ 0);

      int fboId = GlUtil.createFboForTexture(inputTexId);
      GlUtil.focusFramebufferUsingCurrentContext(fboId, width, height);
      GLES20.glReadPixels(
          /* x= */ 0,
          /* y= */ 0,
          width,
          height,
          GLES20.GL_RGBA,
          GLES20.GL_UNSIGNED_BYTE,
          pixelBuffer);
      GlUtil.checkGlError();
      bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
      bitmap.copyPixelsFromBuffer(pixelBuffer);

3. Process bitmap in TFLite
4. Pass the processed output image

      texId =
          GlUtil.createTexture(
              outputBitmap.getWidth(),
              outputBitmap.getHeight(),
              /* useHighPrecisionColorComponents= */ false);
      GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texId);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_NEAREST);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S, GL10.GL_REPEAT);
      GLES20.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT);
      GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, /* level= */ 0, outputBitmap, /* border= */ 0);

5. Pass it to your glProgram
   glProgram.setSamplerTexIdUniform("uTexSampler", texId, /* texUnitIndex= */ 0);

6. Use setVideoEffects in ExoPlayer to set your custom effect

Here is just a high level overview of what needs to be done

[dhruvsingla273]

Thanks for the response
I will try this approach and let you know.

Meanwhile
If I can get a steady stream of a Bitmaps or ByteBuffer with my final frame data (using a custom decoder from MediaCodec),
can I somehow use ExoPlayer to render them using the player?
Means I won't be having all the frames at a single time, I can have like say 2 sec of frames and then continue processing other frames and keep passing to ExoPlayer to render.

Do you have idea how can we do this?

[droid-girl]

@tonihei could you help with the last question?

[tonihei]

That's not possible at the moment because there is no mode in which you can render existing decoded frames from ByteBuffers. ExoPlayer supports image playback, as a series of Bitmaps, but this sounds very inefficient as you have to move the fully decoded buffers around. I think it's usually preferable to do all processing on the GPU? I think that's the approach @droid-girl explained above.

[dhruvsingla273]

Hi @droid-girl, using the effect method I was able to get the bitmap at each frame,
But I am not able to render anything on the surface.
So my surface is coming as black.
I even tried just passing the as it is bitmap without any change to the surface but it is not rendering.

Any idea why?

[droid-girl]

It is hard to say without looking at the code and debugging it.
Please check if you have any GL errors and if you correctly bind processed frames

[dhruvsingla273]

So, i was experimenting with the .setVideoEffect
I am able to render the original video.
Whenever I use my model it takes some milli seconds to run

And then the player crashes/stops at sometimes after 10/4/5 sec of playback

The code:
`
@OverRide
public void drawFrame(int inputTexId, long presentationTimeUs)
throws VideoFrameProcessingException {
try {
long time = System.currentTimeMillis();

        int fboId = GlUtil.createFboForTexture(inputTexId);
        GlUtil.focusFramebufferUsingCurrentContext(fboId, width, height);
        GLES20.glFinish();

        GLES20.glReadPixels(0, 0, width, height, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, pixelBuffer);

        GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, fboId);
        GLES20.glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0, GLES20.GL_TEXTURE_2D, 3, 0);
        GlUtil.checkGlError();

        Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
        bitmap.copyPixelsFromBuffer(pixelBuffer);

        output = model(bitmap, interpreter);
        Bitmap bitmap1 = output.getBitmap();
        pixelBuffer.position(0);

        int texId = GlUtil.createTexture(outputwidth, outputheight, false);
        GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, texId);
        GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap1, 0);

        glProgram.use();
        glProgram.setSamplerTexIdUniform("uTexSampler", texId, 0);
        glProgram.bindAttributesAndUniforms();
        GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
        GlUtil.checkGlError();
        System.out.println("save + read time for each frame with model " + (System.currentTimeMillis() - time));

    } catch (GlUtil.GlException e) {
        throw new VideoFrameProcessingException(e, presentationTimeUs);
    }
}`

This is the error log I get:
at androidx.media3.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:1112) at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:544) at android.os.Handler.dispatchMessage(Handler.java:102)  at android.os.Looper.loop(Looper.java:255)  at android.os.HandlerThread.run(HandlerThread.java:67)  Caused by: android.media.MediaCodec$CodecException: Error 0xfffffff3 at android.media.MediaCodec.releaseOutputBuffer(Native Method) at android.media.MediaCodec.releaseOutputBufferInternal(MediaCodec.java:3568) at android.media.MediaCodec.releaseOutputBuffer(MediaCodec.java:3542) at androidx.media3.exoplayer.mediacodec.SynchronousMediaCodecAdapter.releaseOutputBuffer(SynchronousMediaCodecAdapter.java:163) at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.renderOutputBufferV21(MediaCodecVideoRenderer.java:1666) at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.renderOutputBuffer(MediaCodecVideoRenderer.java:1627) at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.processOutputBuffer(MediaCodecVideoRenderer.java:1356) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:2010) at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:827) at androidx.media3.exoplayer.video.MediaCodecVideoRenderer.render(MediaCodecVideoRenderer.java:940)  at androidx.media3.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:1112)  at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:544)  at android.os.Handler.dispatchMessage(Handler.java:102)  at android.os.Looper.loop(Looper.java:255)  at android.os.HandlerThread.run(HandlerThread.java:67) 

[dhruvsingla273]

Lets say it took 2 sec to process 10 frames,
so the progress bar of exoplayer (which displays the time) goes forward by 2 sec, when it should have stayed at 0 itself because not all the 60 frames for 2 sec are processed.

Can you help with this

[halx99]

Current ExoPlayer not support ByteBuffer mode, but it's possible to patch MediaVideoCodecRenderer.java: https://github.com/axmolengine/axmol/blob/dev/core/platform/android/java/src/org/axmol/lib/MediaCodecVideoRenderer.java#L16

[dhruvsingla273]

@halx99 I already tried this approach, I am able to get a image there (have to set the surface to null to make it run in ByteBuffer mode)
But then I am unable to render the image/bitmap back to the surface because there is no surface

Do you have any solution for this?

[halx99]

You can render the bytebuffer(NV12 pixel data) to GLSurface with custom fragment shader, refer to: https://github.com/axmolengine/axmol/blob/dev/core/renderer/shaders/videoTextureNV12.frag

Edit: in some old device the decoded bytebuffer would be I420 pixel data, refer axmolengine/axmol#2050

[dhruvsingla273]

Okay will try that,

Does anyone knows in which format the buffer is stored when the encoding is mkv

[halx99]

Okay will try that,

Does anyone knows in which format the buffer is stored when the encoding is mkv

I guess bytebuffer mode always NV12, refer https://github.com/axmolengine/axmol/wiki/Media-Player

[dhruvsingla273]

@halx99 I tried to get the buffer at the processOutputBuffer function in MediaCodecVideoRenderer
So i set the surface to null and then get the bitmap.

But after the player tries to display the frame as there is no surface it is unable to render,
then the same buffer keeps coming in the processOutputBuffer.

[droid-girl]

We are planning to merge a TFLite sample to platform-samples. Take a look for more details on the integration here

[dhruvsingla273]

Thanks
I will surely take a look

[droid-girl]

Thanks. Please update if you have any questions.

[google-oss-bot]

Hey @dhruvsingla273. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@dhruvsingla273 if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.