Shaders applied as GLEffects in transformer have different results than applying them directly to a GLSurfaceView

[kleiren]

I am applying shaders to a GLSurfaceView where an Exoplayer is playing a video (based on your VideoProcessingGLSurfaceView demo), and then I am exporting the video with Transformer applying the same shaders to the MediaItems (as GLEffects).

The effects are notably different in the GLSurfaceView and in the Transformer result, it seems the Transformer has more "intensity" in the effects.

As an example, I have a simple brightness shader as a GLEffect:
fragment_shader_brightness.glsl

[...]
void main() {
    highp vec4 textureColor = texture2D(uTexSampler, vTexSamplingCoord);
    gl_FragColor = vec4((textureColor.rgb + vec3(intensity)), textureColor.w);
}

For an intensity of 0,2 to a black background, the results are the following:

GLSurfaceView	Transformer

The shader applied in the Transformer is clearly more "white". Getting closer to 100 intensity the results are more similar, however in lower intensities the shaders differ a lot. This happens for every shader I am using (in some is more noticeable than others).

I know I could apply the effects to the Exoplayer through setVideoEffects(), but it does not work well when changing the effect intensity "in real time" while the video is playing, and I need it to be applied to pictures as well. (An interactive preview of the effect applied to the Transformer result)

My question is, is there something special done to the textures when applying the effects? How can I replicate the same shader result applying it to a GLSurfaceView.

[dway123]

Summary: I'd recommend you to simply use setVideoEffects. More detailed answer below

is there something special done to the textures when applying the effects? How can I replicate the same shader result applying it to a GLSurfaceView.

It seems you're using 2 different paths. (1) ExoPlayer's VideoProcessingGLSurfaceView is a very thin wrapper with ~no color transformations (operating on input colors by default), whereas (2) Transformer, or ExoPlayer.setVideoEffects does have some color transformations (operating in linear colors by default). The "something special" is these color transformations, in our glsl shaders. If you're only doing SDR, you can apply the SMPTE 170m transfers (EOTF / OETF) to replicate this, but this may differ based on input/output colors if you have HDR input, so I'd recommend you to simply use setVideoEffects if possible to have consistent color transformations applied, especially as you're using this in Transformer too.

I know I could apply the effects to the Exoplayer through setVideoEffects(), but it does not work well when changing the effect intensity "in real time" while the video is playing, and I need it to be applied to pictures as well. (An interactive preview of the effect applied to the Transformer result)

You should be able to call setVideoEffects while the video is playing, to update the effects. We hope to improve this implementation in the future though as it does a lot of reconfiguration at the moment under the hood, but I believe this should already work. We unfortunately don't properly handle updating the displayed frame when setVideoEffects is called while a video is paused, but we do plan to support this in the future.

[kleiren]

Thanks for the quick response!

This is exactly the problem I am having. Using setVideoEffects() directly on the exoPlayer the results are the same.

As you say, the "default" shader I am applying to the GLSurfaceView does not apply any color transformation before applying other shaders, but the "DefaultShaderProgram" from media3 applies color transformations and then the rest of the shaders.

Using setVideoEffects() is the solution. However, I have a couple of issues with it:

• For some reason it does not work for 9/16 videos (Black Exoplayer when using setVideoEffects() inside a fragment with a constrained Layout parent #791), easy to work around.
• It is slow when many shaders are applied, specially with complex shaders (my use case allows users to update effect intensity interactively with a slider)
• If a couple of effects are set too fast, the video freezes without a log and does not recover until recreated (as above, allowing the user to set the effects with a slider sends tons of "setVideoEffects()" consecutively.

I will play around to see if I can make setVideoEffects() work for me or if I go the "color transformation in my implementation" route.

[dway123]

Thanks, this makes sense now. Yeah I suspect that the right path here for our library would be to fix the issues you mentioned with setVideoEffects (especially as we plan for this to be how apps should see effects on ExoPlayer). I think this specific bug is working as intended, since we don't intend to add color transformations to GlSurfaceView.

Could you please consider filing new issues for your second and third bullet points? Also, some info before you file in case it helps...

It is slow when many shaders are applied, specially with complex shaders (my use case allows users to update effect intensity interactively with a slider)

I wonder if the issue is that setVideoEffects is called whenever the slider moves, since it does incur some background processing, and maybe a debounce (calling setVideoEffects a max of once per 10ms or 100ms) might help. We currently flush all frames and re-create all shaders when it's called, but we hope to improve this in the future (ex. to re-create only updated shaders).

If a couple of effects are set too fast, the video freezes without a log and does not recover until recreated (as above, allowing the user to set the effects with a slider sends tons of "setVideoEffects()" consecutively.

This sounds like you're hitting a hang (deadlock??) somehow. Please file a bug for this since this definitely shouldn't happen.

[dway123]

Closed per the prior comment, but please file new bugs for the other issues mentioned in the bullet points (or reopen if you think there's something else we should have covered)

[claincly]

For updating your effects responsively with a slider - I've tried the following and it's working well (the effect updates smoothly on slider value change), but it has some caveats that I'll list towards the bottom

• Make an API to your GlEffect to be able to update the effect parameter (like brightness) to the corresponding GlShaderProgram. Currently, a GlEffect is stateless and is like a factory for the shader program, you need to make GlEffect stateful here

• When the GlShaderProgram receives the parameter update, set the uniform (one variant be GlUtil.setIntUniform) on the actual glsl to update the shader

• The updated effect should apply on the next frame.

Caveats

• This is not currently officially supported, so it might stop working some time
• When paused, updating the effect has no effect (pun!). Meaning the user wouldn't be able to see the updated effect until they play again
• It only works when your list of effects doesn't change. That is, it works when you only need to change the parameters of the effects that are already in the list of effects.

[andrewlewis]

@claincly Did you have a prototype for allowing triggering re-rendering (and showing the resulting frame as quickly as possible)? I remember we talked about that a long time ago. Maybe we should mark this as an enhancement request to implement that.

[kleiren]

Nice, I'll play around with this info.
In my implementation, changing the slider recreated the full GLEffect with a new value (intensity) for the shader to use, and sent the effect list to the video by setVideoEffects(). I need the option to add and remove multiple GLEffects and I would like for it to be updated when the video is paused. As I said, when adding and changing effect values quickly, specially with lut shaders and a vignette shader, the video freezes. I will look into it a bit more.

Regarding the previous info about the color changes, I have been looking into the source code of the DefaultShaderProgram and realized that for SDR content I can disable "enableColorTransfers".
For my use case, this is enough for now, and it is a simple change in the Transformer creation:

val videoProcessor =  DefaultVideoFrameProcessor.Factory.Builder().setEnableColorTransfers(false).build()
Transformer.Builder(context).setVideoFrameProcessorFactory(videoProcessor) [...]

This way the color transfers are disabled and the shaders are applied the same way I am applying them in the GLSurfaceView. This is also useful to apply the same effects to other stuff that is not a video, as I am using the player and transformer to generate videos with images as sources, and as far as I know images are not being affected by setVideoEffects() on the exoplayer for now (images are not even shown). I am using an ImageOutput and applying the effects to them on their own.

Thanks for the help!

[kleiren]

Update:

The last comment "solution" does not work if you want to apply OverlayEffects to the resulting video. For some reason, the OverlayEffect always converts the bitmaps to optical color and expects the video to be this way when applying the shader. This basically makes the colors of the overlay wrong when color transfers are disabled in the transformer, because it only affects the video and not the overlayEffect.

#1095