Migrate rendering plugins to `SurfaceProducer` after stable release

[matanlurey]

tl;dr:

• It is both non-trivial to support the Android SurfaceTexture API in Impeller (on Android with Vulkan)
• SurfaceTexture is not a desirable integration point (based on talking to the Android team directly)
• Usage is quite low, and there is an easy/mechanical path to migrate it to something better

Background

The Flutter engine on Android supports two external rendering sources, SurfaceTexture¹²³ (an OpenGLES texture) and ImageReader⁴⁵⁶ (a GPU-ready buffer). Image.getHardwareBuffer, which is used by our ImageReader-based code path, is only supported on newer Android API versions (API level 28).

In Impeller, for the first time, we'll support Vulkan, starting at Android API version 29, over OpenGLES (all Flutter Android apps currently use Skia, which in turn uses OpenGLES). Unfortunately, it's non-trivial to use an OpenGLES texture (SurfaceTexture) in Vulkan, and after some internal discussion with the Android team, we've come to the conclusion that SurfaceTexture is not the right integration point in general.

I did a quick scan of usage of (create|register)SurfaceTexture, both internally and externally:

• Internal numbers are small (single digits)
• Within org: flutter, we have about 17 usages
• Github-wide, I see <400 usages, many of which are ancient projects, forks, or otherwise not critical ecosystem pieces

Proposal

We'll add a new, parallel API/extension point, i.e. (create|register)Surface⁷. Similar to our platform view mechanics, we'll transparently either use SurfaceTexture (on older devices) or ImageReader (on newer devices), and end-users (users that consume plugins that use (create|register)Surface will not have to do or configure anything.

Timeline

• 1. Add a new API, (create|register)Surface, which delegates (internally) to SurfaceTexture or ImageReader
• 2. Migrate internal plugins
• 3. Migrate org: flutter plugins and examples
• 4. Add user-facing documentation on flutter.dev about the recommended approach
• 5. Deprecate (create|register)SurfaceTexture. In Impeller, issue a warning at runtime if it is used (Proposal: Decide what parts of the Java embedder rendering APIs to deprecate #148557)
• 6. In Impeller, issue an error if it is used (Done: Issue anERROR instead of an INFO for a non-working API. engine#52892)
• 7. By the time Skia-Ganesh backend is removed, remove (create|register)SurfaceTexture entirely

Testing Strategy

Our ability to run comprehensive Android device tests in the engine repository is very limited (as of 2023-12-06):

• Android engine unit tests are mock-heavy
• Separation of framework and engine means to properly write an integration test you need code on "both" sides

@johnmccutchan coincidentally has a napkin-math proposal to improve the state of Android platform testing, which is critical for stability of our Android platform views and plugins. As necessary I plan to pair with him to iron out that proposal, land it, and then consume whatever infrastructure we built out/refactor out to also test our (create|register)Surface API.

Some scenarios will include:

• An older Android {device, emulator} that will implicitly use surface textures
• A newer Android {device, emulator} that will implicitly use hardware buffers (via ImageReader)
• Try dynamically resizing the texture at runtime
• Try "animated" textures (i.e. each frame is a different image)
• Try tearing down and re-setting up views that use external textures

---

/cc @dnfield @jonahwilliams @gaaclarke @johnmccutchan @reidbaker

Footnotes

1. android.graphics.SurfaceTexture ↩

2. surface_texture_external_texture.h ↩

3. platform_view_android.cc ↩

4. android.media.ImageReader ↩

5. image_external_texture.h ↩

6. platform_view_android.cc ↩

7. android.view.Surface ↩

[jonahwilliams]

FYI @stuartmorgan

I think this is a reasonable proposal, the only caveat I would consider is how exactly we do the migration for the flutter owned packages and plugins. I think we have a generic policy of trying to support the current and last stable Flutter verisons, which on the surface (haha) would prevent us from starting the migration until after the next stable. Though Its possible that there is some compile time trickery and/or reflection we could do in Java to be backwards compatible.