Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Impeller] Implement YUV texture import and sampling for video player frames. #50730

Merged
merged 1 commit into from
Mar 5, 2024

Conversation

chinmaygarde
Copy link
Member

@chinmaygarde chinmaygarde commented Feb 16, 2024

This patch rounds out support for importing Android Hardware Buffers as Vulkan
images and sampling from them.

Not all Android Hardware Buffers are in formats that Vulkan (let alone Impeller)
understands. Some YUV textures have formats with no equivalent vk::Format enum
and need extra information on how to sample from them.

This patch adds support for correctly importing and sampling from such formats.

There are severe restrictions on how sampling from such external formats works.
For one, it isn’t possible to assign a combined image sampler in the render
pass. The pipeline itself needs to be rebuilt to reference a specially created
immutable sampler. This immutable sampler is a combination of the usual
information present in an Impeller SamplerDescriptor as well as a custom
“conversion” object obtained while importing the Android Hardware Buffer.

There is no way to predict what conversion object will be necessary ahead of
time as this will depend on the source of the Android Hardware Buffers and is
likely different for different video feeds, camera sources, and other Android
Hardware Buffer texture sources.

To handle this uncertainty, a new pipeline variant with a JIT determined
immutable sampler will be hashed and cached before being used in a render pass.
The number of pipeline variants created just-in-time will depend on the number
of sampler variants used in the render pass to sample from wrapped Image. For
instance, specifying a sampler with a different address mode will likely result
in a new pipeline variant being created.

In most cases however, there will just be one or two additional pipeline
variants per application. Impellers sampler diversity is very low with most
samplers being the usual NN samplers. It may be possible to preload even this
pipeline by trying known conversions. As said previously, there can only be a
handful of these conversions.

More restrictions on sampling from such images includes being limited to
VK_FILTER_LINEAR without additional format and extension wrangling and
performance penalties.

Fixes flutter/flutter#142082.

@chinmaygarde chinmaygarde added Work in progress (WIP) Not ready (yet) for review! e: impeller labels Feb 16, 2024
@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact "@test-exemption-reviewer" in the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@chinmaygarde
Copy link
Member Author

I'm working with the video_player example in the linked issue with a local patch in flutter/flutter#142082 (comment).

@chinmaygarde
Copy link
Member Author

This is good for a review. I've editing the original comment with the commit message and rounded out documentation and cleaned up logs. There are no validation errors but I need to manually check more applications than just the video player still.

Copy link
Member

@jonahwilliams jonahwilliams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like to make the LRU cache effective for video streams we need to increase the cache size, for me 36 did it. Not important to handle right now.

Overall LGTM!

@chinmaygarde
Copy link
Member Author

@jonahwilliams @johnmccutchan An issue that I haven't investigated yet is the following log emitted every frame. It doesn't seem to cause issues but it was confusing.

acquireLatestImage image's fence was never signalled.

After backgrounding, the log changes to:

Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers

over and over.

LMK if you have ideas, but this was happening before and after my patch so I'm treating it as unrelated.

@jonahwilliams
Copy link
Member

I noticed that too: flutter/flutter#143720 . We should follow up with Android folks to figure out if we're misusing the aquireFence API, because its actually surprisingly expensive too.

@flutter-dashboard
Copy link

Golden file changes have been found for this pull request. Click here to view and triage (e.g. because this is an intentional change).

If you are still iterating on this change and are not ready to resolve the images on the Flutter Gold dashboard, consider marking this PR as a draft pull request above. You will still be able to view image results on the dashboard, commenting will be silenced, and the check will not try to resolve itself until marked ready for review.

Changes reported for pull request #50730 at sha b6b69d0

@flutter-dashboard
Copy link

Golden file changes are available for triage from new commit, Click here to view.

Changes reported for pull request #50730 at sha f40adec

Copy link
Member

@gaaclarke gaaclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, we should have tests though. At a minimum a golden test that renders a YUV texture would be good. I was wondering if just writing out a camera frame as raw bytes might be the easiest thing to get the input?

#include "impeller/renderer/pipeline.h"

namespace impeller {

class SamplerVK;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@chinmaygarde chinmaygarde Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -16,53 +16,114 @@

namespace impeller {

/// Abstract base class that represents a vkImage and an vkImageView.
class YUVConversionVK;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

//----------------------------------------------------------------------------
/// @brief When sampling from textures whose formats are not known to
/// Vulkan, a custom conversion is necessary to setup custom
/// samplers. This accessor provides this conversion is one is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// samplers. This accessor provides this conversion is one is
/// samplers. This accessor provides this conversion if one is

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

/// @brief Determines if swapchain image. That is, an image used as the
/// root render target.
///
/// @return Whether or not this is a swapchain image.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// @return Whether or not this is a swapchain image.
/// @return `true` if this is a swapchain image.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't author these docstrings. Just move them to use headerdoc annotations. We can clean up the actual strings in a later patch.

@@ -15,6 +15,8 @@

namespace impeller {

class SamplerVK;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

YUVConversionVK& operator=(const YUVConversionVK&) = delete;

//----------------------------------------------------------------------------
/// @return If this conversion is valid for use with images and samplers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// @return If this conversion is valid for use with images and samplers.
/// @return `true` if this conversion is valid for use with images and samplers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines 66 to 67
/// @return The descriptor.
///
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// @return The descriptor.
///

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

const YUVConversionDescriptorVK& rhs) const;
};

class SamplerVK;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I also noticed that the Vulkan backend specific object ImmutableSamplerKey was missing the VK suffix. Patched that up too.

@chinmaygarde
Copy link
Member Author

a golden test that renders a YUV texture

That isn't possible because the golden test harness doesn't support VK_ANDROID_external_memory_android_hardware_buffer and VK_KHR_sampler_ycbcr_conversion. So even if I could capture a frame and create a wrapped image (which I can't since there is no mechanism to create AHBs today), there'd be no way to render it with Vulkan without those extensions.

I wanted to bring testing this during the weekly to see if the Android Plugins Test/Test-Infra workstream has ideas on this. Because if I were to rely on existing tests for verifying camera feeds, I suspect there aren't any.

@jonahwilliams
Copy link
Member

We'll need to test this with the scenario app harness.

@johnmccutchan
Copy link
Contributor

a golden test that renders a YUV texture

That isn't possible because the golden test harness doesn't support VK_ANDROID_external_memory_android_hardware_buffer and VK_KHR_sampler_ycbcr_conversion. So even if I could capture a frame and create a wrapped image (which I can't since there is no mechanism to create AHBs today),

FYI- You can create AHBs using the NDK:
https://developer.android.com/ndk/reference/group/a-hardware-buffer#ahardwarebuffer_allocate

@johnmccutchan
Copy link
Contributor

@jonahwilliams @johnmccutchan An issue that I haven't investigated yet is the following log emitted every frame. It doesn't seem to cause issues but it was confusing.

acquireLatestImage image's fence was never signalled.

This seems specific to video decoders as other producers do signal the fence. I've asked Android folks about this.

After backgrounding, the log changes to:

By backgrounding do you mean backgrounding the app or ?

@chinmaygarde
Copy link
Member Author

By backgrounding do you mean backgrounding the app or...

Yeah. Just backgrounding the app.

@chinmaygarde
Copy link
Member Author

chinmaygarde commented Mar 4, 2024

You're not causing the stall. Though I still can't explain the results of the logs pulled from the bots as described in flutter/flutter#144352. Going to investigate this further today.

@chinmaygarde
Copy link
Member Author

My mid-day update of struggles with the scenarios app.

Screenshot 2024-03-04 at 10 40 21 AM
  • I was facing the same hang as @dnfield was when using an arm64 emulator on my M1 Mac when running any Vulkan application (not just Impeller). The guidance from an internal team was that the emulator version was too old (32) and it needed to be updated to at least (34). @dnfield is adding a doctor check for this in Update the emulator binary included in the engine checkout flutter#144561.
  • Unfortunately, updated to emulator version 34 turned the hang into a crash. This has been reported to the emulator team in b/327501571 (see the updates towards the end).

My plan of action for the rest of the day.

  • I'm going to investigate the minor pixel differences reported in the non external textures untriaged digests. My running theory is that using YUV conversions for non external texture formats is not benign, and we are using the qualified sampling rules when i.e, eNearest vs eLinear using the external which is causing the differences.
  • I am going to see if validations are running on the bots to see if I can get any information (in case @matanlurey already knows). I get no validation errors locally so this is a bit of a long shot since the layers are the same.

ExternalTextureTests_testMediaSurface

@chinmaygarde chinmaygarde force-pushed the ahb_vulkan_textures branch 2 times, most recently from d6e52bb to dd87020 Compare March 4, 2024 21:35
@flutter-dashboard
Copy link

Golden file changes are available for triage from new commit, Click here to view.

Changes reported for pull request #50730 at sha dd87020

@chinmaygarde
Copy link
Member Author

I'm going to investigate the minor pixel differences reported in the non external textures untriaged digests. My running theory is that using YUV conversions for non external texture formats is not benign, and we are using the qualified sampling rules when i.e, eNearest vs eLinear using the external which is causing the differences.

This is now fixed!

The latest golden file changes now show the differences @matanlurey was running into when he manually pulled the images.

Screenshot 2024-03-04 at 2 09 29 PM

@dnfield I trying to run this on his emulator.

@matanlurey
Copy link
Contributor

The latest golden file changes now show the differences @matanlurey was running into when he manually pulled the images.

Woohoo!

@dnfield
Copy link
Contributor

dnfield commented Mar 4, 2024

I was able to get this running on an emulator by disabling vulkan validation layers locally.

I'm reproducing the color incorrectness on my emulator, but not on a physical Pixel 7 device.

@chinmaygarde
Copy link
Member Author

End of day update:

  • I haven't been able to reproduce any rendering issues on any physical device except the PowerVR GE8320 whose issues are unrelated.
  • Any emulator that has Vulkan validations enabled hangs (versions < 34) or crashes (version 34) (the emulator itself, not just the process). This is tracked in b/327501571.
  • I haven't been able to get even the Vulkan Capabilities Viewer app running on the emulators without crashing. There are even more reduced test case in the linked Buganizer that are problematic.
  • On devices, I have verified that the application runs without any validation errors.
  • The minor pixel differences that were reported by the gold instance have were fixed today.

I am completely out of ideas as even non-Flutter applications and reduced test cases are not running correctly on the emulators. In the interest of making progress, I would like us to merge this patch with either with a gold instance running the tests on an actual device with a followup based on the resolution of b/327501571, or triaging the current digests as positive (ugh).

If there is a non-Flutter Vulkan app running with AHB running on the emulators, or if we can reproduce this issue on any device, I'd love to see it. But I recommend now we treat the emulator issue separately.

@matanlurey
Copy link
Contributor

I would like us to merge this patch with either with a gold instance running the tests on an actual device with a followup based on the resolution of b/327501571, or triaging the current digests as positive (ugh).

So, strictly speaking, even the current digest is better than a blank image, in terms of correctness.

I don't think it's feasible for us to have a pre-submit with an actual device, or if it is, we'll need to divorce that option from landing this PR. It's clear to me you put a lot of time into this, and this gets us to a better place, so I'm OK with merging as positive - I can make a note in the README that the emulator output of the Vulkan tests doesn't (quite) match reality.

Wdut @gaaclarke @jonahwilliams?

@jonahwilliams
Copy link
Member

As long as we have issues tracking the follow up work and some coverage of this (even if partially working), then landing SGTM.

@matanlurey
Copy link
Contributor

Yeah good point, +1 tracking issue - and I do think it should be a P1, it will be hard to catch regressions on Vulkan if we can't trust the output image.

Copy link
Member

@gaaclarke gaaclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for all the work to get this tested, lgtm

@chinmaygarde
Copy link
Member Author

I marked the digests as "positive". Filed an issue to track the emulator instability. Dan closed in as a duplicate of his tracking issue. It is P1 and cross-referenced with the internal bug. Separately, I filed flutter/flutter#144632 to track backing out the digests marked incorrectly positive.

Landing this.

@chinmaygarde chinmaygarde added the autosubmit Merge PR when tree becomes green via auto submit App label Mar 5, 2024
@chinmaygarde
Copy link
Member Author

Hmm, I marked the digests as positive 35 mins ago but the gold check is still pending. Anyway, the triage page is clean. I'm landing now.

@jonahwilliams
Copy link
Member

Auto submit won't work. I'd just hit the update branch button and let it go again.

… frames.

This patch rounds out support for importing Android Hardware Buffers as Vulkan
images and sampling from them.

Not Android Hardware Buffers are in formats that Vulkan (let alone Impeller)
understands. Some YUV textures have formats with no equivalent `vk::Format` enum
and need extra information on how to sample from them.

This patch adds support for correctly importing and sampling from such formats.

There are severe restrictions on how sampling from such external formats works.
For one, it isn’t possible to assign a combined image sampler in the render
pass. The pipeline itself needs to be rebuilt to reference a specially created
immutable sampler. This immutable sampler is a combination of the usual
information present in an Impeller SamplerDescriptor as well as a custom
“conversion” object obtained while importing the Android Hardware Buffer.

There is no way to predict what conversion object will be necessary ahead of
time as this will depend on the source of the Android Hardware Buffers and is
likely different for different video feeds, camera sources, and other Android
Hardware Buffer texture sources.

To handle this uncertainty, a new pipeline variant with a JIT determined
immutable sampler will be hashed and cached before being used in a render pass.
The number of pipeline variants created just-in-time will depend on the number
of sampler variants used in the render pass to sample from wrapped Image. For
instance, specifying a sampler with a different address mode will likely result
in a new pipeline variant being created.

In most cases however, there will just be one or two additional pipeline
variants per application. Impellers sampler diversity is very low with most
samplers being the usual NN samplers. It may be possible to preload even this
pipeline by trying known conversions. As said previously, there can only be a
handful of these conversions.

More restrictions on sampling from such images includes being limited to
`VK_FILTER_LINEAR` without additional format and extension wrangling and
performance penalties.

Fixes flutter/flutter#142082.
@flutter-dashboard
Copy link

Golden file changes are available for triage from new commit, Click here to view.

Changes reported for pull request #50730 at sha 6ef67ae

@chinmaygarde
Copy link
Member Author

Thats odd, I wonder why Gold missed this one in the earlier run.

Screenshot 2024-03-05 at 12 41 18 PM

@chinmaygarde chinmaygarde merged commit 6594bca into flutter:main Mar 5, 2024
26 of 27 checks passed
@chinmaygarde chinmaygarde deleted the ahb_vulkan_textures branch March 5, 2024 20:42
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 5, 2024
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Mar 6, 2024
auto-submit bot pushed a commit to flutter/flutter that referenced this pull request Mar 6, 2024
…144653)

flutter/engine@49bc157...effcf97

2024-03-05 98614782+auto-submit[bot]@users.noreply.github.com Reverts "[Impeller] Enable depth buffer clipping & Stencil-then-Cover path rendering. (#51209)" (flutter/engine#51217)
2024-03-05 30870216+gaaclarke@users.noreply.github.com [Impeller] implement blur style solid (flutter/engine#50892)
2024-03-05 xilaizhang@google.com [g3 roll] Revert "Remove unused drone_dimension field" (flutter/engine#51214)
2024-03-05 skia-flutter-autoroll@skia.org Roll Skia from 17677914dabf to 37947aec8c5c (5 revisions) (flutter/engine#51211)
2024-03-05 737941+loic-sharma@users.noreply.github.com Fix git hooks on Windows (flutter/engine#51203)
2024-03-05 bdero@google.com [Impeller] Enable depth buffer clipping & Stencil-then-Cover path rendering. (flutter/engine#51209)
2024-03-05 bdero@google.com [Impeller] Fix convex triangulation winding bug for multi-contour paths. (flutter/engine#51198)
2024-03-05 skia-flutter-autoroll@skia.org Roll Skia from 875b356e4d96 to 17677914dabf (2 revisions) (flutter/engine#51207)
2024-03-05 gspencergoog@users.noreply.github.com Skip configuration dependency if unit tests are disabled (flutter/engine#51179)
2024-03-05 chinmaygarde@google.com [Impeller] Implement YUV texture import and sampling for video player frames. (flutter/engine#50730)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC rmistry@google.com,zra@google.com on the revert to ensure that a human
is aware of the problem.

To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autosubmit Merge PR when tree becomes green via auto submit App e: impeller platform-android will affect goldens
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Impeller] Support Vulkan YUV texture sampling for composition of video player frames.
6 participants