Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flutter should use Metal instead of OpenGL on iOS #18208

Closed
eseidelGoogle opened this issue Jun 5, 2018 · 28 comments
Closed

Flutter should use Metal instead of OpenGL on iOS #18208

eseidelGoogle opened this issue Jun 5, 2018 · 28 comments

Comments

@eseidelGoogle
Copy link
Contributor

@eseidelGoogle eseidelGoogle commented Jun 5, 2018

From #33079:

Based on guidance from the Skia team, the Metal backend is ready for integration into Flutter. This is a meta-bug listing the various tasks that need to be undertaken for the switch to be completed.

The Skia Metal backend is currently only available on iOS versions 11.0 and above. It is unlikely that the minimum supported version will be lowered all the way down to 8.0 (the current minimum supported by Flutter). The recommended approach is to compile both the OpenGL and Metal backends in the iOS engine and use runtime checks to only use the Metal backend on versions 11.0 and above. OpenGL will still be used on older iOS versions till support for the same persists.

Tasks

Tracked on this project board but the major ones are:

  • Add the enable-metal GN flag to existing builds of the iOS engine and re-baseline benchmarks that measure binary size. Binary sizes are expected to regress by around 220 Kbytes uncompressed.
  • Enable the Metal backend on iOS versions 11.0 and above.
  • Add an enable-metal shell flag that allows overriding the rendering backend to use without needing version checks. Duplicate all existing benchmarks to run the same with the Metal backend in addition to the OpenGL backend.
  • Ensure plugins that that provide pixel buffers to to the engine for inline composition have the appropriate compatibility flags set.
  • Ensure platform views are wired up.

References:


This is really a Skia bug, but tracking it here since we've gotten a flood of questions about this after Apple's announcement this week of deprecating OpenGL.

My understanding is Skia has an experimental Metal backend, but it would take some work to bring it to production ready.

Skia also already has a Vulkan backend which Flutter uses on Fuchsia, and has historically used on some Android phones (and could use again if there was demand -- it was recently disabled due to lack of demand).

Another advantage of moving to Metal sooner rather than later is that I'm told that Apple's GL drivers appear to be less maintained than they used to be, and Flutter/Skia have seen a resulting up-tick in Apple-specific GL bugs.

@eseidelGoogle
Copy link
Contributor Author

@eseidelGoogle eseidelGoogle commented Jun 6, 2018

Filed a bug with Skia https://bugs.chromium.org/p/skia/issues/detail?id=8050 and have reached out to the team directly. There is some non-trivial amount of work to do here, but certainly possible.

@amrsolimanmoustafa
Copy link

@amrsolimanmoustafa amrsolimanmoustafa commented Jun 6, 2018

I am now developing my first mobile app with flutter when i publish it apple store through couple of week what will happen ?

@pvmhoang
Copy link

@pvmhoang pvmhoang commented Jun 6, 2018

I hope Flutter Dev Team will give us the answer about this one soon. I have just started to learning Flutter :(

@ghost
Copy link

@ghost ghost commented Jun 6, 2018

This is great news. Really happy about the transparency the flutter team are giving to events and users concern.

Vulcan and metal are not that dissimilar and there is a project called Molten that allowed Vulcan code to work at runtime on metal apple. I am sure the skis team are on to it.

Anyway this maybe will open the door to 3d scenegraphs but I digress .

@eseidelGoogle
Copy link
Contributor Author

@eseidelGoogle eseidelGoogle commented Jun 6, 2018

Apple has not announced any removal of OpenGL support. Flutter works great on iOS, including iOS 12.

@ghost
Copy link

@ghost ghost commented Jun 7, 2018

looks like Molten is already in the build of skia !
https://skia.googlesource.com/skia/+/master/BUILD.gn

SO fantastic ...

@Chozzle
Copy link

@Chozzle Chozzle commented Jul 27, 2018

I think improvements need to be made quickly on the graphics rendering pipeline. This is the biggest thing holding our company back from developing in flutter. Look at the Hamilton app on an iPhone 6 for an example of unacceptable performance.

@Chozzle
Copy link

@Chozzle Chozzle commented Aug 8, 2018

Any update on using MoltenVK? Does anyone know if this would allow much 2d performance improvement?

@SAGARSURI
Copy link

@SAGARSURI SAGARSURI commented Sep 9, 2018

Even I am eagerly looking for an update on this from the dev team.

@eseidelGoogle
Copy link
Contributor Author

@eseidelGoogle eseidelGoogle commented Sep 10, 2018

The bug to track is in Skia: https://bugs.chromium.org/p/skia/issues/detail?id=8050. Once Skia has support for Metal, Flutter's Engine can also add support. There likely won't be any updates to this bug in Flutter until the Skia issue is resolved. We speak with Skia team members regularly and they're aware of the ask.

Flutter works great on iOS today. However, we expect Flutter may work even better on iOS with Metal in the long-term (as I'm told OpenGL drivers on iOS are less well maintained than Metal ones).

@FullstackJack
Copy link

@FullstackJack FullstackJack commented Mar 1, 2019

Any movement on this? Should we start worrying that Skia won't make the June 2019 deadline causing Flutter apps to not work on iOS 13? That Skia ticket doesn't seem to have any progress attached.

@eseidelGoogle
Copy link
Contributor Author

@eseidelGoogle eseidelGoogle commented Mar 1, 2019

I'm not aware of any June 2019 deadline. Can you link to some communication from Apple?

@ott-web
Copy link

@ott-web ott-web commented Mar 3, 2019

My mistake. I had read speculation that it would be dropped with new versions of iOS and tvOS at WWDC 2019 in June. I just found this documentation though. See "Deprecations" section. They have begun phasing out OpenGL, but in your experience how many versions will Apple continue to support a deprecated API? We're already one in.

https://developer.apple.com/tvos/whats-new/

@stuartmorgan
Copy link
Contributor

@stuartmorgan stuartmorgan commented Mar 3, 2019

Carbon was deprecated in 2012, and it's not until the next release of macOS that it will finally be unavailable (as a side effect of dropping 32-bit support).

There's no way of predicting for any given API how long exactly Apple will keep it around, but announcing the deprecation of a widely used API and then suddenly removing it shortly after is not historically how they have done things. It would also be unusual for them to completely remove a major API without announcing the removal (not just deprecation) in advance.

@eseidelGoogle
Copy link
Contributor Author

@eseidelGoogle eseidelGoogle commented May 1, 2019

For those following along at home, progress continues on Metal support in Skia as part of https://bugs.chromium.org/p/skia/issues/detail?id=8243. Still a lot of work to do here, just letting folks know this is not forgotten. :)

@tvolkert
Copy link
Contributor

@tvolkert tvolkert commented May 6, 2019

Update from @bsalomon on #31086 (comment)

it's probably not too early to start trying to integrate the Metal backend. We're at the point where it draws almost everything correctly and performance is approx equal to GL with low hanging fruit still left to pick.

@chinmaygarde
Copy link
Member

@chinmaygarde chinmaygarde commented May 11, 2019

Support in Flutter for rendering using Metal on iOS landed in flutter/engine#8936. For now, the engine has to be manually built with the --enable-metal flag to have this backend enabled. We need to work some more on lowering the minimum iOS version. Currently, enabling the Metal backend bumps the floor up to 11.0 (from 8.0). But, I don't believe all the latest APIs are strictly necessary.

@stuartmorgan stuartmorgan mentioned this issue May 12, 2019
3 of 3 tasks complete
@lightandshadow68
Copy link

@lightandshadow68 lightandshadow68 commented Aug 14, 2019

Apple hasn't removed OpenGLES on iOS, despite being depricated, because there are a significant number of existing iOS apps that have shipped which are still using it. However, Apple's new Catalyst environment (iOS on Mac / Marzipan) does not support OpenGL ES as no Catalyst apps have shipped.

@stuartmorgan
Copy link
Contributor

@stuartmorgan stuartmorgan commented Aug 14, 2019

There are currently no specific plans to support Catalyst in Flutter, so that's not really a factor in near-term prioritization of this issue.

@Hixie
Copy link
Member

@Hixie Hixie commented Oct 22, 2019

We were discussing this in today's Triage meeting. Our current plan is to turn Metal on soon. This will increase the size of our engine by ~200KB. We can eventually reduce this by removing the OpenGL code but (a) that would drop support for <iOS10, which we don't want to do anytime soon (Skia only does Metal from iOS10), and (b) would require some work from the Skia team (OpenGL can't currently be removed from the codebase).

@stuartmorgan
Copy link
Contributor

@stuartmorgan stuartmorgan commented Oct 22, 2019

that would drop support for <iOS10

Also the iPhone 5 and 5c, and the 4th generation iPad, all of which can run iOS 10, but don't support Metal.

@jmagman jmagman added this to Awaiting triage in Mobile - iOS engine review Feb 25, 2020
@fisforfaheem

This comment was marked as off-topic.

@dannyvalentesonos
Copy link
Contributor

@dannyvalentesonos dannyvalentesonos commented Mar 27, 2020

We were discussing this in today's Triage meeting. Our current plan is to turn Metal on soon. This will increase the size of our engine by ~200KB. We can eventually reduce this by removing the OpenGL code but (a) that would drop support for <iOS10, which we don't want to do anytime soon (Skia only does Metal from iOS10), and (b) would require some work from the Skia team (OpenGL can't currently be removed from the codebase).

Hi @Hixie . Is there an update on when Flutter will be using Metal?
Also, once it is turned on, will plugins like the video_player which use the Texture Registry work immediately?

Thanks!

@chinmaygarde
Copy link
Member

@chinmaygarde chinmaygarde commented Mar 27, 2020

@dannyvalentesonos: The March 2020 milestone is accurate. Metal is currently feature complete on iOS and gated behind a flag. Will provide updated guidance in a couple of days.

will plugins like the video_player which use the Texture Registry work immediately?

Yes, plugins that use the texture registry like the video player and the camera are tested to work. More guidance surrounding  applications that use the texture registry to provide texture created using OpenGL for inline composition in Flutter will be forthcoming.

@chinmaygarde chinmaygarde moved this from To Do to In Progress in Metal Renderer Mar 31, 2020
chinmaygarde added a commit to chinmaygarde/flutter_engine that referenced this issue Mar 31, 2020
Fixes flutter/flutter#18208. All Metal for iOS related work items described in https://github.com/orgs/flutter/projects/5 have been completed.
@chinmaygarde
Copy link
Member

@chinmaygarde chinmaygarde commented Apr 1, 2020

I am landing flutter/engine#17431 which turns Metal on-by-default on iOS. A Metal on iOS FAQ Wiki page attempts to answer all questions pertaining to this change.

Metal Renderer automation moved this from In Progress to Done Apr 1, 2020
Mobile - iOS engine review automation moved this from Awaiting triage to Engineer reviewed Apr 1, 2020
chinmaygarde added a commit to flutter/engine that referenced this issue Apr 1, 2020
Fixes flutter/flutter#18208. All Metal for iOS related work items described in https://github.com/orgs/flutter/projects/5 have been completed.
@lock
Copy link

@lock lock bot commented Apr 15, 2020

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@lock lock bot locked and limited conversation to collaborators Apr 15, 2020
goderbauer added a commit to goderbauer/engine that referenced this issue Apr 16, 2020
Fixes flutter/flutter#18208. All Metal for iOS related work items described in https://github.com/orgs/flutter/projects/5 have been completed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Mobile - iOS engine review
  
Engineer reviewed
Metal Renderer
  
To Do
Linked pull requests

Successfully merging a pull request may close this issue.