Support FlutterView

[bbrto21]

There are internal requirements for FlutterView.

The main purpose is to provide some interface for each language(C#/C++) that works as a UI component of Tizen rather than a single page FlutterApp. (see : https://docs.flutter.dev/development/add-to-app)
Although specific requirements have not yet been communicated, we may need to support both. however, our top priority is to provide Flutter-tizen as a NUI component.

I think there are two ways to achieve this. the first way is to provide it by implementing it in embeddings, and the other way is to provide it by implementing in Tizen platform level after including flutter-tizen to Tizen platform like the webview provided in Tizen UI Toolkit. each way has distinct pros and cons I think but I'm approaching it the first way.

Several issues have already been opened and reviewed, but it seems to me that they all serve the same purpose.
so I will close these issues in favor of managing them as a single issue. (@JSUYA agreed to this)

• FlutterView prototype (with EFL EvasGL) #325
• How to load [abi:cxx11] symbol engine#253
• Return Image handle instead of window (EFL) engine#233

Here's a rough to-do list.
(Please feel free to add a specific to do item. honestly, I still don't know exactly what to do 😅)

• Support FlutterView on C#
  • Support FlutterView working on NUI
  • Support FlutterView working on ElmSharp
• Support FlutterView on C++
  • Support FlutterView working on Elementary
• Refactor and Add new APIs
  • The Flutter-tizen API should be reinforced to fit the concept of engine, window, and view.
    It means that the existing API name or behavior can be changed.
• Document the API usage.
  • An end-to-end guide (something like this).
  • How to debug Dart code of the running view (DevTools support).

[bbrto21]

Things that have been revealed about NUI

• NUI is based on dali2 from Tizen 6.0 and later. and currently, we cannot use dali2 in our engine build configuration. see this
• NUIApplication can be used to provide a embedding. Window of NUIAapplication is based on wl2 and can obtain a native handle from Window instance, so we don't have to create a wl2 window ourselves on engine side for single page application scenario.
  • So I think it's better generally to pass the render target to the engine on the embedding side rather than creating the render target in the engine. ※ a render target means a material for creating an egl surface, such as a wl2/elm window or evas image object.
  • This requires a change in the flutter-tizen API and embedding.
• There is an idea(from @JSUYA ) to show the rendering results of Flutter-tizen through ImaveView.
  • The following is a brief explanation.
    1. First, create a dali texture using tbm and get a resource URL from it.
    2. Create an eglsurface using tbm.
    3. Set the resource URL to ImaveView

I also tried this method, but I couldn't see anything through ImageView. I use egl display generated from the tbm dummy display, and the egl surfaces created from these were used without errors while rendering. I wonder what I'm missing in this method.

[bbrto21]

I have succeeded in showing the rendering result of flutter through ImageView with the idea mentioned above.
My mistake was that I didn't fix getting the geometry from the window in the engine initialization flow, so I didn't tell flutter the correct size.

by the way for some reason... I can't attach any image files. Is it because I am in the office?

[xuelian-bai]

I have a question, on which scenario will flutterview be used on Tizen? like hybrid app on android: NUI page-->flutter page-->NUI page? or display flutter app as an UI component on NUI app?

[bbrto21]

There are no clear requirements related to the scenario.
But basically, I think both scenarios should be supported. and I'm focusing on implementing the latter scenario.
since flutter can monopolize Window in the former scenario, I think it's an easy scenario to support on current flutter-tizen

[xuelian-bai]

ok, I just can't see any advantages of the latter scenario. Apps can use NUI components directly, why use flutter's UI components? If there are multiple components, how to handle events like focus?
Does this scenario like NUI webview?

[JSUYA]

I have a question, on which scenario will flutterview be used on Tizen? like hybrid app on android: NUI page-->flutter page-->NUI page? or display flutter app as an UI component on NUI app?

Hi
Although it is not an official request, there was a request to use flutter in the previously written NUI app and EFL app from VD.

This feature follows Flutter's Add to app concept.

The first of the two scenarios you talked about was
https://docs.flutter.dev/development/add-to-app/android/add-flutter-fragment
The second is this.
https://docs.flutter.dev/development/add-to-app/android/add-flutter-view

Since there is no concept of fragment in tizen, I think that the FlutterView we are planning now can replace it.

The event, focus, and a11y have not been considered yet.
We still don't know everything about NUI (DALi) and we are now at POC level. :)

[bbrto21]

@JSUYA
I think the first scenario is no different from the full view app using multi-window.
(For example, one of the two windows is used for the Tizen UI toolkit, and the other one is for Flutter.)
and In my opinion, Fragments in Android are similar to Tizen's component-based app .

However, the app models of the two platforms did not correspond exactly, I tried to relate it as much as possible.

[bbrto21]

I have succeeded in showing the rendering result of flutter through ImageView with the idea mentioned above.

[update 22/4/7]
Here is a sample image of my PoC, but for some reason, the rendering stops in the middle. I still don't know why.

I confirmed through PoC development that flutter can be shown as below based on NUI's ImageView and NativeImageQueue.

However, I think this approach requires too much implementation(touch/key event handlers, IME and etc) on embedding...
So I'm going to check out the following ideas.

1. Create Native Flutter View using the DALI API on engine side(Native side).
2. Send handle of Native Flutter View to the embedding(C# side).
3. Create NUI FlutterView based on View or CustomView using the given handle in the embedding.
4. Use it on Application based on NUIApplication(a.k.a FlutterNUIAppication)

That's it.
After that, I would like to discuss the implement direction of FlutterView.

If you have any ideas or questions for a FlutterView implementation, please let me know.

[updates 22-08-03]
here is sample code that is part of this PoC. In this sample, to minimize the engine-side implementation, the NativeImageQueue is created on the C# side, and its raw handle is passed to the engine.

[bbrto21]

Support FlutterView working on NUISupport FlutterView working on NUI

To achieve this, we considered using Tizen 6.5 packages for making sysroot to build engine.
however, I found that the engine built using this sysroot can not use on Tizen 5.5 and earlier version due to glibc version issue.
also flutter-tizen are guarantees to support the Tizen version as below, so the sysroot using the Tizen 6.5 packages is not available in the current state.

Supported devices

• Galaxy Watch: Tizen 4.0 (2018) or later
• Smart TV: Tizen 6.0 (2021) or later (older models are not supported due to security reasons)
• IoT (RPi 3/4): Tizen 5.5 (2020) or later

glibc version

• Tizen 6.0 or higher : glibc-2.30
• Tizen 5.5 or lower : glibc-2.24

If you want to build engine with Tizen 6.5 packages, you may need below.
this is a problem that occurs because the version of the atk included in the Tizen package increases. (cc. @wanchao-xu)

• -Wno-c++11-narrowing
• --allow-deprecated-api-calls // gn flag

[bbrto21]

here is my attempt to enable rtti and disable use_flutter_cxx and upgrade Tizen to 6.5

• engine: bbrto21/engine@1a156ab
• buildroot: bbrto21/buildroot@fdeb7ab
• tizen_tools: bbrto21/tizen_tools@e91025b

If you don't need to disable use_flutter_cxx then I guess engine's changes that including header files explicitly is not required.
here is minimum changes to enable rtti I think.

[JSUYA]

buildroot: bbrto21/buildroot@fdeb7ab

There are two cases so far that use_flutter_cxx disable is needed.
(When using Dali::AnyCast(), when using std::string GetUrl())

When AnyCast() uses std::type_info and typeid() inside a function, types with the same name built in different libcxx conflicts.
In our case, the type check is not essential as this type is clear. And since we can access Impl class(public), I was able to solve it through a direct call as shown below.

// Link Error
//tbm_surface_queue_ = Dali::AnyCast<tbm_surface_queue_h>(dali_native_image_queue_->GetNativeImageSourceQueue());  

tbm_surface_queue_ =
        static_cast<Dali::Any::AnyContainerImpl<tbm_surface_queue_h>*>(
            dali_native_image_queue_->GetNativeImageSourceQueue().mContainer)
            ->GetValue();

The std::string case is a std::__cxx11::string problem of std::string similar to the above.
In order to set the tbm_surface_queue_h on the Image(target) that Flutter renders, we need to use the std::string GetUrl() API.
@bbrto21 found a way to get Url from NUI (C#) code to solve this problem.

NativeImageQueue queue = new NativeImageQueue((uint)imageView.Size2D.Width, (uint)imageView.Size2D.Height, NativeImageQueue.ColorFormat.RGBA8888);
Type typ = typeof(NativeImageQueue).BaseType.BaseType.BaseType;
FieldInfo type = typ.GetField("swigCPtr", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance);
System.Runtime.InteropServices.HandleRef value = (System.Runtime.InteropServices.HandleRef)type?.GetValue(queue);

RenderTarget = value.Handle; //RenderTarget is member of FlutterDesktopWindowProperties

With these two workarounds I don't think use_flutter_cxx disable is necessary.

I think the remaining issue is the KeyEvent.(Of course there may be more.)
If we implement KeyEvent callback in FlutterEngine, we will face std::string problem when getting KeyString().
In my expectation, it should to be implemented in flutter-tizen by providing APIs such as FlutterDesktopViewOnTouch and FlutterDesktopViewOnKey.
I'll share it again after a few POCs.

[swift-kim]

I created a wiki page: https://github.com/flutter-tizen/flutter-tizen/wiki/Adding-to-existing-app

@JSUYA The embedder has been migrated not to use use_flutter_cxx. Can the workarounds you mentioned in the above be removed or improved?

[JSUYA]

@JSUYA The embedder has been migrated not to use use_flutter_cxx. Can the workarounds you mentioned in the above be removed or improved?

tbm_surface_queue_h tbm_surface_queue_ =
          Dali::AnyCast<tbm_surface_queue_h>(dali_native_image_queue->GetNativeImageSourceQueue());

The --system-cxx option is required to apply the above code.
Build is fine, but I'm getting a Dali::Exception in Runtime.

E/ConsoleMessage(10460): terminate called after throwing an instance of 'Dali::DaliException'
E/ConsoleMessage(10460): onSigabrt called
..
E/DALI    (10460): Any::Get(). Trying to retrieve a value of a different type than the template one.

In api-version 6.5 builds, we can't change this code unless --system-cxx is default.

I don't think we need to change NativeImageQueue specifically because we don't necessarily need to use the Dali API.(at least for now)
and
NUI NativeImageQueue code has been improved in #444 (comment)

[swift-kim]

@JSUYA

we can't change this code unless --system-cxx is default.

--system-cxx is already the default build option for the 6.5 embedders as I mentioned.

I don't think we need to change NativeImageQueue specifically because we don't necessarily need to use the Dali API.

I thought the code could be moved to the embedder (C++) side because you can now use the GetUrl API. Isn't it the case?

In order to set the tbm_surface_queue_h on the Image(target) that Flutter renders, we need to use the std::string GetUrl() API.
@bbrto21 found a way to get Url from NUI (C#) code to solve this problem.

[JSUYA]

@swift-kim

--system-cxx is already the default build option for the 6.5 embedders as I mentioned.

Um... what I mean is that if we set --api-version to 6.5 then system-cxx should be applied by default. I misunderstood. I will patch soon.

I thought the code could be moved to the embedder (C++) side because you can now use the GetUrl API. Isn't it the case?

As you said, we can create Dali's NativeImageQueue and call SetImage() from embedder(C++) code.
But since we still need to pass the ImageView's HandleRef to the embedder, the code to get the HandleRef should remain the same. (GetFieldValue<T>).
What I was worried about was these codes.
Type typ = typeof(NativeImageQueue).BaseType.BaseType.BaseType;
Type imageViewBaseType = typeof(ImageView).BaseType.BaseType.BaseType.BaseType;

we don't necessarily have to move it. Isn't it? now, it is only a difference in where NativeImageQueue is created.
um.. I think that Embedder (Dali) can handle the event callback using ImageView's HandleRef and clean up some of the engine's API .I don't think we need to move the NativeImageQueue and use GetUrl() unless otherwise.