Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. #6456

matanlurey · 2024-04-03T20:12:24Z

WIP: We do not plan to land this PR until the next stable release (>= April 3rd 2024).

Details

Migrates uses of createSurfaceTexture to createSurfaceProducer, which is intended to have no change in behavior, but does change the backend rendering path, so it will require more testing (and we're also open to minor API renames or changes before it becomes stable).

Background

Android plugins previously requested a SurfaceTexture from the Android embedder, and used that to produce a Surface to render external textures on (i.e. video_player). This worked because 100% of Flutter applications on Android used OpenGLES (via our Skia backend), and SurfaceTexture is actually an (opaque) OpenGLES-texture.

Starting soon (roughly ~Q3, this is not a guarantee and just an estimate), Flutter on Android will start to use our new Impeller graphics backend, which on newer devices (>= API_VERSION_28), will default to the Vulkan, not OpenGLES. In other words, SurfaceTexture will cease to work (it is possible, but non-trivial, to map an OpenGLES texture over to Vulkan).

After consultation with the Android team, they helped us understand that vending SurfaceTexture (the consumer-side API) was never the right abstraction, and we should have been vending the producer-side API, or Surface directly. The new SurfaceProducer API is exactly that - it generates a Surface, and similar to our platform view strategy, picks the "right" consumer-side implementation details for the user/plugin packages.

The new SurfaceProducer API has 2 possible rendering types (as an implementation detail):

SurfaceTexture, for older OpenGLES devices, which works exactly as it does today.
ImageReader, for newer OpenGLES or Vulkan devices.

These are some subtle nuances in how these two APIs work differently (one example: flutter/flutter#144407), but our theory at this point is we don't expect these changes to be observed by any users, and we have other ideas if necessary.

Note

These invariants are tested on CI in flutter/engine.

Points of contact:

@matanlurey or @jonahwilliams (Flutter Engine)
@johnmccutchan or @reidbaker (Flutter on Android)

johnmccutchan

woo hoo

…e-producer

matanlurey · 2024-05-16T21:49:48Z

Due to a release hiccup, the stable branch (3.22) is temporarily ahead of the master branch (3.22.0-35.0.pre.34), leading to CI failures (see discord discussion). I've merged back into HEAD and updated the pubspec/CHANGELOG - the rest might be done by someone else while I'm on vacation for a couple of weeks.

johnmccutchan · 2024-05-20T19:52:49Z

...r/video_player_android/android/src/main/java/io/flutter/plugins/videoplayer/VideoPlayer.java

@@ -186,7 +186,7 @@ public void onCancel(Object o) {
          }
        });

-    surface = new Surface(textureEntry.surfaceTexture());
+    surface = surfaceProducer.getSurface();


Note, the returned Surface may become invalid after a suspend / resume. Is there a place in the plugin that will be notified when we resume? We should update the video surface in that callback.

Need to look at this still.

I've made this activity aware. It now tears down the exoplayer on pause and recreates on reusme.

jonahwilliams · 2024-05-24T19:49:29Z

...r/video_player_android/android/src/main/java/io/flutter/plugins/videoplayer/VideoPlayer.java

@@ -48,16 +48,16 @@ final class VideoPlayer {

  private ExoPlayer exoPlayer;

-  private Surface surface;


don't need to cache surface.

jonahwilliams · 2024-05-24T19:50:07Z

...r/video_player_android/android/src/main/java/io/flutter/plugins/videoplayer/VideoPlayer.java

+  }
+
+  public void pauseSurface() {
+    eventChannel.setStreamHandler(null);


I tried just calling exoPlayer.setSurface again with a new surface, however the old surface being disposed seems to break some internal state, so I have to recreate the exo instance. This might imply some loss of state.

I agree but I'm not sure because I'm not deeply familiar with ExoPlayer. Does the case where a video is playing and then the app is moved to the background and brought back get impacted by this, i.e. the video is no longer playing from the point it was stopped?

Let me investigate this .

I looked at this and it seems like the video state is reset. I'm going to see if there is a way to restore it.

I added logic to restore all of the visible exo state when re-creating exo player.

camsim99

LGTM!

johnmccutchan · 2024-05-29T21:56:20Z

@jonahwilliams Just flagging that if we decide to back out our fix that forces ImageReaders to be recreated, we should revert parts of the PR.

…eProducer`. (flutter/packages#6456)

flutter/packages@31d3329...910fabb 2024-05-29 58529443+srujzs@users.noreply.github.com Amend package:web tweaks to allow package:web roll (flutter/packages#6793) 2024-05-29 matanlurey@users.noreply.github.com Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6456) 2024-05-29 joonas.kerttula@codemate.com [google_maps_flutter] Undeprecate BitmapDescriptor methods (flutter/packages#6832) 2024-05-29 matanlurey@users.noreply.github.com Migrate CameraX from SurfaceTexture to SurfaceProducer. (flutter/packages#6462) 2024-05-29 matanlurey@users.noreply.github.com Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6461) 2024-05-29 katelovett@google.com [dynamic_layouts] Remove the dynamic_layouts package (flutter/packages#6830) 2024-05-29 43054281+camsim99@users.noreply.github.com [camerax] Add notes about Android permissions (flutter/packages#6741) 2024-05-29 43054281+camsim99@users.noreply.github.com [Re-land] Bump legacy all_packages project AGP version to 7.0.0, Gradle version to 7.0.2 (flutter/packages#6742) 2024-05-29 engine-flutter-autoroll@skia.org Roll Flutter from a1a33e6 to c85fa6a (20 revisions) (flutter/packages#6829) 2024-05-29 katelovett@google.com [rfw] Migrate deprecated doc references (flutter/packages#6744) 2024-05-29 katelovett@google.com [flutter_adaptive_scaffold] Migrate MaterialStateProperty to WidgetStateProperty (flutter/packages#6743) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…eProducer`." (#6882) Reverts #6456

…#6908) Part of flutter/flutter#148417. I'm working on re-landing #6456, this time without using the `ActivityAware` interface (see flutter/flutter#148417). As part of that work, I'll need to better control the `ExoPlayer` lifecycle and save/restore internal state. These are some proposed refactors to limit how much work `VideoPlayer` is doing, so I can better understand what needs to be reset (or not) internally. Specifically, `VideoPlayer` no longer knows what an `EventChannel` or `EventSink` is, and does not need to manage the lifecycle (it stores a `private final VideoPlayerCallbacks` instead), and instead there is a `VideoPlayerCallbacks` interface that does all that. I'm totally open to: - Landing this as-is (+/- nits) and making minor improvements in follow-up PRs - Making more significant changes to this PR and then landing it - Not landing this PR at all because it doesn't follow the approach the folks who maintain the plugin prefer Also happy to chat in VC/person about any of the changes.

…`. (#6922) Similar to #6908, as part of flutter/flutter#148417. I'm working on re-landing #6456, this time without using the `ActivityAware` interface (see flutter/flutter#148417). As part of that work, I'll need to better control the `ExoPlayer` lifecycle and save/restore internal state. In this PR, I've removed the concept of the class being "initialized" or not - the only thing "initialized" means is "for a given instance of `ExoPlayer`, has received the `'initialized'` event. As a result I removed the quasi-public API that was used for testing only and replaced it with observing what the real production instance does (`Player.STATE_READY`). After this PR, I'll likely do one more pass around the constructors - the constructor that takes an `ExoPlayer` that is marked `@VisibleForTesting` _also_ doesn't make sense once we'll support suspending/resuming video players, so it will need to get reworked (probably into taking a factory method).

@matanlurey

…r`. (flutter#6456) _**WIP**: We do not plan to land this PR until the next stable release (>= April 3rd 2024)_. Work towards flutter/flutter#145930. ## Details Migrates uses of `createSurfaceTexture` to `createSurfaceProducer`, which is intended to have no change in behavior, but _does_ change the backend rendering path, so it will require more testing (and we're also open to minor API renames or changes before it becomes stable). ## Background Android plugins previously requested a `SurfaceTexture` from the Android embedder, and used that to produce a `Surface` to render external textures on (i.e. `video_player`). This worked because 100% of Flutter applications on Android used OpenGLES (via our Skia backend), and `SurfaceTexture` is actually an (opaque) OpenGLES-texture. Starting soon (roughly ~Q3, this is not a guarantee and just an estimate), Flutter on Android will start to use our new Impeller graphics backend, which on newer devices (`>= API_VERSION_28`), will default to the Vulkan, _not_ OpenGLES. In other words, `SurfaceTexture` will cease to work (it is possible, but non-trivial, to map an OpenGLES texture over to Vulkan). After consultation with the Android team, they helped us understand that vending `SurfaceTexture` (the _consumer-side_ API) was never the right abstraction, and we should have been vending the _producer-side_ API, or `Surface` directly. The new `SurfaceProducer` API is exactly that - it generates a `Surface`, and similar to our platform view strategy, picks the "right" _consumer-side_ implementation details _for_ the user/plugin packages. The new `SurfaceProducer` API has 2 possible rendering types (as an implementation detail): - `SurfaceTexture`, for older OpenGLES devices, which works exactly as it does today. - `ImageReader`, for newer OpenGLES _or_ Vulkan devices. These are some subtle nuances in how these two APIs work differently (one example: flutter/flutter#144407), but our theory at this point is we don't expect these changes to be observed by any users, and we have other ideas if necessary. > [!NOTE] > These invariants are [tested on CI in `flutter/engine`](https://github.com/flutter/engine/tree/main/testing/scenario_app/android#ci-configuration). Points of contact: - @matanlurey or @jonahwilliams (Flutter Engine) - @johnmccutchan or @reidbaker (Flutter on Android)

…eProducer`." (flutter#6882) Reverts flutter#6456

…flutter#6908) Part of flutter/flutter#148417. I'm working on re-landing flutter#6456, this time without using the `ActivityAware` interface (see flutter/flutter#148417). As part of that work, I'll need to better control the `ExoPlayer` lifecycle and save/restore internal state. These are some proposed refactors to limit how much work `VideoPlayer` is doing, so I can better understand what needs to be reset (or not) internally. Specifically, `VideoPlayer` no longer knows what an `EventChannel` or `EventSink` is, and does not need to manage the lifecycle (it stores a `private final VideoPlayerCallbacks` instead), and instead there is a `VideoPlayerCallbacks` interface that does all that. I'm totally open to: - Landing this as-is (+/- nits) and making minor improvements in follow-up PRs - Making more significant changes to this PR and then landing it - Not landing this PR at all because it doesn't follow the approach the folks who maintain the plugin prefer Also happy to chat in VC/person about any of the changes.

…`. (flutter#6922) Similar to flutter#6908, as part of flutter/flutter#148417. I'm working on re-landing flutter#6456, this time without using the `ActivityAware` interface (see flutter/flutter#148417). As part of that work, I'll need to better control the `ExoPlayer` lifecycle and save/restore internal state. In this PR, I've removed the concept of the class being "initialized" or not - the only thing "initialized" means is "for a given instance of `ExoPlayer`, has received the `'initialized'` event. As a result I removed the quasi-public API that was used for testing only and replaced it with observing what the real production instance does (`Player.STATE_READY`). After this PR, I'll likely do one more pass around the constructors - the constructor that takes an `ExoPlayer` that is marked `@VisibleForTesting` _also_ doesn't make sense once we'll support suspending/resuming video players, so it will need to get reworked (probably into taking a factory method).

…allback`. (#6982) I'm working on re-landing #6456, this time without using the `ActivityAware` interface (see flutter/flutter#148417). As part of that work, I'll need to better control the `ExoPlayer` lifecycle and save/restore internal state. Follows the patterns of some of the previous PRs, i.e. - #6922 - #6908 The changes in this PR are _mostly_ tests, it was extremely difficult to just add more tests to the already very leaky `VideoPlayer` abstraction which had lots of `@VisibleForTesting` methods and other "holes" to observe state. This PR removes all of that, and adds test coverage where it was missing. Namely it: - Adds a new class, `VideoAsset`, that builds and configures the media that `ExoPlayer` uses. - Removes all "testing" state from `VidePlayer`, keeping it nearly immutable. - Added tests for most of the classes I've added since, which were mostly missing. That being said, this is a large change. I'm happy to sit down with either of you and walk through it. --- Opening as a draft for the moment, since there is a pubspec change needing I want to handle first.

…allback`. (flutter#6982) I'm working on re-landing flutter#6456, this time without using the `ActivityAware` interface (see flutter/flutter#148417). As part of that work, I'll need to better control the `ExoPlayer` lifecycle and save/restore internal state. Follows the patterns of some of the previous PRs, i.e. - flutter#6922 - flutter#6908 The changes in this PR are _mostly_ tests, it was extremely difficult to just add more tests to the already very leaky `VideoPlayer` abstraction which had lots of `@VisibleForTesting` methods and other "holes" to observe state. This PR removes all of that, and adds test coverage where it was missing. Namely it: - Adds a new class, `VideoAsset`, that builds and configures the media that `ExoPlayer` uses. - Removes all "testing" state from `VidePlayer`, keeping it nearly immutable. - Added tests for most of the classes I've added since, which were mostly missing. That being said, this is a large change. I'm happy to sit down with either of you and walk through it. --- Opening as a draft for the moment, since there is a pubspec change needing I want to handle first.

@jonahwilliams

…`setCallback` for suspend/resume lifecycles. (#6989) _ð��« BLOCKED: I guess this can't land until the new API makes it into stable in a week or two?_ --- Effectively enough towards flutter/flutter#148417, but we still need to document it on flutter.dev. This is the last _technical_ PR I'll work on towards the plugin work (assuming we don't find additional bugs/issues). /cc @jonahwilliams @chinmaygarde @johnmccutchan.

…this time with `setCallback` for suspend/resume lifecycles. (flutter#6989)" This reverts commit 62b4cb0.

… with setCallback for suspend/resume lifecycles" (#7497) Reverts #6989 Fixes flutter/flutter#154054 , flutter/flutter#154050

…ucer, this time with setCallback for suspend/resume lifecycles" (flutter#7497)" This reverts commit f61a98a.

Migrate video_player/android from SurfaceTexture->SurfaceProducer.

77ccaea

matanlurey requested a review from johnmccutchan April 3, 2024 20:12

github-actions bot added platform-android p: video_player labels Apr 3, 2024

johnmccutchan approved these changes Apr 3, 2024

View reviewed changes

matanlurey added 4 commits May 16, 2024 08:57

Merge branch 'main' into video-player-surface-producer

54b24cb

Update pubspec and CHANGELOG.

6866d44

Merge remote-tracking branch 'upstream/main' into video-player-surfac…

d3eacba

…e-producer

++

782ee27

matanlurey marked this pull request as ready for review May 16, 2024 19:50

matanlurey requested a review from camsim99 as a code owner May 16, 2024 19:50

matanlurey assigned jonahwilliams May 16, 2024

matanlurey mentioned this pull request May 16, 2024

Stage changes to flutter/packages as a PR that migrates to SurfaceProducer flutter/flutter#145930

Closed

jonahwilliams mentioned this pull request May 19, 2024

Video Player does not disable image on Android with Impeller enabled flutter/flutter#148647

Closed

johnmccutchan approved these changes May 20, 2024

View reviewed changes

johnmccutchan reviewed May 20, 2024

View reviewed changes

jonahwilliams added 5 commits May 24, 2024 09:50

Merge branch 'main' into video-player-surface-producer

3ae8f75

formatting

89296f6

++

c0c6b9b

make activity aware.

8b46bd6

Merge branch 'main' into video-player-surface-producer

9b378ae

jonahwilliams reviewed May 24, 2024

View reviewed changes

jonahwilliams requested a review from johnmccutchan May 24, 2024 19:50

formatting.

54a0738

github-actions bot added the p: camera label May 24, 2024

jonahwilliams added 3 commits May 24, 2024 13:12

pubspec

713cbc7

revert changes to camera

6664ecd

++

76ec4bd

jonahwilliams requested a review from camsim99 May 29, 2024 19:28

camsim99 approved these changes May 29, 2024

View reviewed changes

jonahwilliams added the autosubmit Merge PR when tree becomes green via auto submit App label May 29, 2024

auto-submit bot merged commit a5dd314 into main May 29, 2024
74 checks passed

auto-submit bot deleted the video-player-surface-producer branch May 29, 2024 21:18

engine-flutter-autoroll mentioned this pull request May 30, 2024

Roll Packages from 31d332900d98 to 910fabb4886b (11 revisions) flutter/flutter#149321

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 30, 2024

a5dd314 Migrate video_player/android from SurfaceTexture->`Surfac…

62cb718

…eProducer`. (flutter/packages#6456)

huycozy mentioned this pull request Jun 3, 2024

VideoPlayer failed to auto-rotate the video after upgrade flutter to version 3.22.0. flutter/flutter#149556

Closed

jonahwilliams mentioned this pull request Jun 6, 2024

Revert "Migrate video_player/android from SurfaceTexture->SurfaceProducer." #6882

Merged

auto-submit bot pushed a commit that referenced this pull request Jun 6, 2024

Revert "Migrate video_player/android from SurfaceTexture->`Surfac…

8a2c4e4

…eProducer`." (#6882) Reverts #6456

matanlurey mentioned this pull request Jun 11, 2024

Refactor VideoPlayer to be less exposed to EventChannel & related #6908

Merged

matanlurey mentioned this pull request Jun 13, 2024

Move Player.Listener impl, remove @VisibleForTesting isInitialized. #6922

Merged

arc-yong pushed a commit to Arctuition/packages-arc that referenced this pull request Jun 14, 2024

Revert "Migrate video_player/android from SurfaceTexture->`Surfac…

7f5b75e

…eProducer`." (flutter#6882) Reverts flutter#6456

matanlurey mentioned this pull request Jun 25, 2024

Final refactor of video_player_android before SurfaceProducer#setCallback. #6982

Merged

reidbaker mentioned this pull request Jul 2, 2024

i151188 update espresso dependencies v2 reidbaker/packages#6

Closed

11 tasks

LinXunFeng added a commit to LinXunFeng/packages that referenced this pull request Aug 26, 2024

Revert "[video_player] Relands flutter#6456: Uses SurfaceProducer, …

43b106a

…this time with `setCallback` for suspend/resume lifecycles. (flutter#6989)" This reverts commit 62b4cb0.

auto-submit bot pushed a commit that referenced this pull request Aug 26, 2024

Revert "[video_player] Relands #6456: Uses SurfaceProducer, this time…

f61a98a

… with setCallback for suspend/resume lifecycles" (#7497) Reverts #6989 Fixes flutter/flutter#154054 , flutter/flutter#154050

AlexV525 mentioned this pull request Sep 6, 2024

[video_player_android] Incorrect orientation when previewing captured videos flutter/flutter#154696

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. #6456

Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. #6456

matanlurey commented Apr 3, 2024

johnmccutchan left a comment

matanlurey commented May 16, 2024

johnmccutchan May 20, 2024

jonahwilliams May 24, 2024

jonahwilliams May 24, 2024

jonahwilliams May 24, 2024

jonahwilliams May 24, 2024

camsim99 May 29, 2024

jonahwilliams May 29, 2024

jonahwilliams May 29, 2024

jonahwilliams May 29, 2024

camsim99 left a comment

johnmccutchan commented May 29, 2024

		@@ -48,16 +48,16 @@ final class VideoPlayer {

		private ExoPlayer exoPlayer;

		private Surface surface;

Migrate video_player/android from SurfaceTexture->SurfaceProducer. #6456

Migrate video_player/android from SurfaceTexture->SurfaceProducer. #6456

Conversation

matanlurey commented Apr 3, 2024

Details

Background

johnmccutchan left a comment

Choose a reason for hiding this comment

matanlurey commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camsim99 left a comment

Choose a reason for hiding this comment

johnmccutchan commented May 29, 2024

Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. #6456

Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. #6456