New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bindings for OpenGL #179

Closed
FlutterIssues opened this Issue Nov 9, 2015 · 28 comments

Comments

Projects
None yet
@FlutterIssues
Copy link

FlutterIssues commented Nov 9, 2015

Issue by unicomp21
Sunday Oct 04, 2015 at 09:59 GMT
Originally opened as https://github.com/flutter/engine/issues/1482


Do we have ES 3.0?

@FlutterIssues

This comment has been minimized.

Copy link

FlutterIssues commented Nov 9, 2015

Comment by abarth
Sunday Oct 04, 2015 at 19:16 GMT


Internally Flutter uses OpenGL to draw, but we don't currently expose OpenGL to authors. Instead, we expose a 2D drawing API via the Canvas interface: http://domokit.github.io/docs/sky/rendering/PaintingCanvas-class.html

@FlutterIssues

This comment has been minimized.

Copy link

FlutterIssues commented Nov 9, 2015

Comment by unicomp21
Sunday Oct 04, 2015 at 20:23 GMT


Are there plans to expose it?

On Sunday, October 4, 2015, Adam Barth notifications@github.com wrote:

Internally Flutter uses OpenGL to draw, but we don't currently expose
OpenGL to authors. Instead, we expose a 2D drawing API via the Canvas
interface:
http://domokit.github.io/docs/sky/rendering/PaintingCanvas-class.html


Reply to this email directly or view it on GitHub
https://github.com/flutter/engine/issues/1482#issuecomment-145379067.

@FlutterIssues

This comment has been minimized.

Copy link

FlutterIssues commented Nov 9, 2015

Comment by abarth
Sunday Oct 04, 2015 at 20:24 GMT


It's something we want to implement in the long term, but we don't have any plans to expose it in the near term.

@abarth abarth changed the title Where's the OpenGL? Add bindings for OpenGL Nov 13, 2015

@Hixie Hixie added this to the Flutter 1.0 milestone Feb 3, 2016

@bp74

This comment has been minimized.

Copy link

bp74 commented Mar 6, 2016

I said this on the Flutter forum earlier, but adding support for OpenGL would bring a whole new audience to Flutter - i think about those developers working with Adobe Air to write games for iOS and Android. Flutter could be a great cross platform development environment for games. The current canvas API is nice for little games but we need to get closer to the metal for more advanced use cases :) Thanks for keeping this under consideration.

@eseidelGoogle

This comment has been minimized.

Copy link
Contributor

eseidelGoogle commented Mar 7, 2016

There are GL bindings for Dart available via dart's dart-ext: native extension mechanism:
https://github.com/google/dart-gl

Flutter's engine doesn't yet support dart-ext:, but we have a bug on file about adding such #2396.

@abarth

This comment has been minimized.

Copy link
Contributor

abarth commented Jun 1, 2016

/cc @apwilson

@ivanpopelyshev

This comment has been minimized.

Copy link

ivanpopelyshev commented Aug 2, 2016

If you implement it, I'll make pixi.js port for flutter.

@sethladd sethladd modified the milestones: Flutter 2.0 or later, Flutter 1.0 Sep 14, 2016

@sethladd

This comment has been minimized.

Copy link
Contributor

sethladd commented Sep 20, 2016

Thanks very much for the report! This isn't on the roadmap right now. It would be cool to see this one day, but it's not something we're taking action on.

We can reopen later.

Thanks for checking out Flutter!

@bergwerf

This comment has been minimized.

Copy link

bergwerf commented Jun 23, 2017

Just stirring this up, how far away is a potential GLES binding? I would love to build an interactive molecule viewer in Dart for Android/iOS.

@unicomp21

This comment has been minimized.

Copy link

unicomp21 commented Sep 29, 2017

Xamarin w/ .net core is building up a lot of steam for android/ios, it even supports AR Kit, OpenGL, etc. Flutter is way behind.

@PaulHMason

This comment has been minimized.

Copy link

PaulHMason commented Jan 30, 2018

This is definitely something that has a lot of value for developers (games on mobile are a bit of a big thing). If the bindings are there then something like Starling, Corona or Phaser are definitely doable.

@eseidelGoogle

This comment has been minimized.

Copy link
Contributor

eseidelGoogle commented Feb 1, 2018

So far our understanding has been that there are already good ways to share OpenGL code between platforms (e.g. C++) without Flutter doing anything special here.

There are a variety of ways I could imagine integrating such shared OpenGL code with other Flutter code (such as using our foreign texture API: https://docs.flutter.io/flutter/widgets/Texture-class.html from a plugin wrapping the c++: https://flutter.io/platform-channels/ to expose the resulting surface from the GL code for compositing with the rest of the flutter scene). Happy to brainstorm further ways if someone would like to try this.

So far exposing all of OpenGL as Dart hasn't seemed worth the complexity trade-off. Flutter's engine-level APIs are intentionally small:
https://docs.flutter.io/flutter/dart-ui/dart-ui-library.html

There is a separate OpenGL-in-Dart library which one could also imagine building a custom flutter/engine build to expose:
https://github.com/google/dart-gl
But again, that library is pretty huge and would be difficult for Flutter to maintain.

I welcome your thoughts.

@matejthetree

This comment has been minimized.

Copy link

matejthetree commented Feb 3, 2018

+1 on this. Gaming is huge and there is bunch of adobe air developers waiting to come to something new. Dart is step up and all we need to expose OpenGL in a way that adobe air does, and someone will male sparrow starling and there we go. Check the community behind starling

@eseidelGoogle

This comment has been minimized.

Copy link
Contributor

eseidelGoogle commented Apr 25, 2018

FYI @mravn-google and @sigurdm who designed the foreign texture API and may be interested to see this. Thanks for the thoughts @luigi-rosso

@MarkOSullivan94

This comment has been minimized.

Copy link

MarkOSullivan94 commented May 13, 2018

@luigi-rosso could you open source that example? Would be curious to see how it would work.

@sethladd is it time to reopen this issue?

Out of interest, would it be possible to wrap ARCore and/or ARKit for doing AR related stuff with Flutter apps? If so, I'd happily make a start on porting ARCore for Flutter.

@hungrymonkey

This comment has been minimized.

Copy link

hungrymonkey commented Jun 5, 2018

apple just depreciated Opengl

https://developer.apple.com/ios/whats-new/#deprecationofopengles

Apps built using OpenGL ES will continue to run in iOS 12, but Open GL ES is deprecated in iOS 12. Games and graphics-intensive apps that previously used OpenGL ES should now adopt Metal.

Metal is designed from the ground up to provide the best access to the modern GPUs on iOS, macOS, and tvOS devices. Metal avoids the overhead inherent in legacy technologies and exposes the latest graphics processing functionality. Unified support for graphics and compute in Metal lets your apps efficiently utilize the latest rendering techniques. For information about developing apps and games using Metal, see the developer documentation for Metal, Metal Performance Shaders, and MetalKit.

@Takhion

This comment has been minimized.

Copy link

Takhion commented Jun 6, 2018

Given the news from Apple, does that mean that Skia could become the new universal, OpenGL-like API for drawing on any platform? Since it has backends for OpenGL, Vulkan e Metal (experimental, but still), perhaps expanding and exposing more functionality in Skia would be a possible solution.

@rock3r

This comment has been minimized.

Copy link
Contributor

rock3r commented Jun 6, 2018

Are you suggesting that Skia should be an abstraction layer over Metal/OpenGL/Vulkan? Would make some sense, but I think it's something better discussed on the Skia project :)

@sprhawk

This comment has been minimized.

Copy link

sprhawk commented Oct 1, 2018

Another big vote for this one! I think having a subset of GL directly exposed to Flutter with Dart would be phenomenal.

The foreign texture api was the easiest way to go about this for a quick test with iOS. I basically did a test of what @eseidelGoogle mentioned in the first part of his comment, just sharing a GL frame buffer rendered by a Flutter plugin.

Engine related:

* There doesn't seem to be engine side access to each of the platform specific foreign texture registries. So I had to do this as a plugin, but it would be interesting to expose a way to grab the registry from the engine side to expose an OpenGL "canvas".

* I think you would ultimately want to do this on the engine side (if you're planning to use GL in Dart) as the tonic bindings seem like they would be more performant than having to expose the OpenGL api via the MethodChannel. I agree it's a big API, but I think a wrapper around the programmable pipeline is really all you need (ES2 to begin with as it has the most overlap with other versions of GL, similar to what WebGL does).

* Context switching issues: doing this as a plugin requires a second GL context, I'm a little wary of having to do that, but I assume there's no good way to get around that without forming some kind of GL PictureRecorder that can run before SKIA operations on the same context (Vulkan implications aside...).

Plugin related:

* Seems like there's a hard requirement to pass a full copy of the CVPixelBuffer for the texture. It would be really nice if both contexts could share the CVPixelBuffer and if Flutter would only call CVOpenGLESTextureCacheCreateTextureFromImage when the returned CVPixelBuffer changes for the registered texture. You could use the CVPixelBuffer as the backing texture for the FrameBuffer in your GL context and as a simple (read only) texture in the SKIA GL context (I think it should be similar in Vulkan). I'm not 100% sure that it'd work, but I think it should. Right now I have to allocate and copy the CVPixelBuffer every frame. Maybe I'm missing something...

* Eventually it'd be nice to be able to specify a set of CVPixelBuffers to properly establish front/back buffers but that would require even further changes to the texture api.

* Here's the simple result of this tinkering:
  ![image1](https://user-images.githubusercontent.com/454182/39166318-da41e1fc-473d-11e8-9913-bfeb67729993.png)

I think the way the iOS copying CVPixelBuffer is far less efficient than Android side implementation.

On Android side, see in flutter-engine/shell/platform/android/android_external_texture_gl.cc, it used a SurfaceTexture to share GL Texture between Android GL context and flutter/skia context (however I don't know SkImage::MakeFromTexture if also very efficient)

On iOS side, see in flutter-engine/shell/platform/ios/ios_external_texture_gl.cc, it used copyPixelBuffer , then CVOpenGLESTextureCacheCreateTextureFromImage.
That is, in my knowledge, it will copy from OpenGL texture to main memory , then copy from main memory to skia GL texture. I think it is far less efficient.

What about doing similar to Android? Or what about flutter engine provides a GL texture resource and context, and call plugin's method to draw something on it ?

Updated:

I saw some articles about CVOpenGLESTextureCacheCreateTextureFromImage , it will make it transfer faster. but I don't know whether it should be created over and over again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment