Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vuforia only render in OpenGLES2, not Metal #126

Open
igrir opened this issue Aug 6, 2019 · 13 comments
Open

Vuforia only render in OpenGLES2, not Metal #126

igrir opened this issue Aug 6, 2019 · 13 comments

Comments

@igrir
Copy link

igrir commented Aug 6, 2019

Hello, I'm having an issue after integrating react-native with Vuforia. Everything runs well, but Vuforia always runs on OpenGLES2 instead of Metal. I really need to run it on Metal because in the app we also need to run the ARKit. I tried to force Graphics API on Unity by change it only to Metal but the result was the same!

image

Here's the log when I am not using react-native-unity-view:

....
Initializing Vuforia...
2019-08-06 13:44:58.044546+0700 unityreact4[975:63447] User pre-granted access to the camera
2019-08-06 13:44:58.044615+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Vuforia selected rendering API Metal
2019-08-06 13:44:58.044632+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Vuforia SDK version 8.3.8
2019-08-06 13:44:58.044646+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Vuforia SDK build 3621
2019-08-06 13:44:58.097371+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:58: Matched precache profile for Apple, iPad7,5, iOS 12, 800.1
2019-08-06 13:44:59.658717+0700 unityreact4[975:63447] INFO/AR(975) 2019-08-06 13:44:59: Vuforia Fusion: Detected and enabled use of ARKit
2019-08-06 13:44:59.660540+0700 unityreact4[975:63447] Rendering mode set to: Metal
2019-08-06 13:44:59.672348+0700 unityreact4[975:63447] Rendering mode set to: Metal
2019-08-06 13:44:59.675556+0700 unityreact4[975:63447] DEBUG/AR(975) UIView has CAMetalLayer layer class
2019-08-06 13:44:59.675786+0700 unityreact4[975:63447] DEBUG/AR(975) UIView does not respond to selector renderFrameVuforia
....

And here's how it's reported after I used react-native-unity-view:

....
Initializing Vuforia...
2019-08-06 14:42:47.621329+0700 unityreact4[1053:77026] User pre-granted access to the camera
2019-08-06 14:42:47.621433+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Vuforia selected rendering API OpenGL ES 2.x
2019-08-06 14:42:47.621476+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Vuforia SDK version 8.3.8
2019-08-06 14:42:47.621504+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Vuforia SDK build 3621
2019-08-06 14:42:47.690246+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:47: Matched precache profile for Apple, iPad7,5, iOS 12, 800.1
2019-08-06 14:42:48.997315+0700 unityreact4[1053:77026] INFO/AR(1053) 2019-08-06 14:42:48: Vuforia Fusion: Detected and enabled use of ARKit
2019-08-06 14:42:48.999518+0700 unityreact4[1053:77026] Rendering mode set to: OpenGL ES 2.0
2019-08-06 14:42:49.036414+0700 unityreact4[1053:77026] Rendering mode set to: OpenGL ES 2.0
2019-08-06 14:42:49.037062+0700 unityreact4[1053:77026] DEBUG/AR(1053) UIView has CAMetalLayer layer class
2019-08-06 14:42:49.037287+0700 unityreact4[1053:77026] DEBUG/AR(1053) UIView does not respond to selector renderFrameVuforia
....

as you can see, it stated as Vuforia selected rendering API OpenGL ES 2.x. It's really strange. Maybe Vuforia detected the react app as OpenGL ES 2? Do I have to set something on XCode to force everything to render on Metal?

Thanks

@zbagley
Copy link

zbagley commented Aug 6, 2019

@igrir As far as I'm aware, this project isn't fully compatible with Metal and is only set up for OpenGL ES 2 as per the docs:

IOS Platform:

Other Settings find the Rendering part, uncheck the Auto Graphics API and select only OpenGLES2.

@igrir
Copy link
Author

igrir commented Aug 6, 2019

@zbagley that's correct, but the thing @f111fei being aware of is the problem mentioned in this issue #29

I successfully included ARKit camera and run it using this library, but unfortunately I can't combine it with Vuforia as it only render black screen. The funny thing is the detection runs well! The 3D object being shown on the position where the marker was, but only the camera won't render

@zbagley
Copy link

zbagley commented Aug 7, 2019

This could very well be a similar error that I've been running into. Ran a minimalist project from scratch (it uses ARFoundation ARKit) and ran into this for the first time when exporting:
image

@igrir
Copy link
Author

igrir commented Aug 7, 2019

@zbagley did you only add OpenGLES2 on your Graphics API? Might be that could be the problem. I've just tried it and also encountered that error.

For my issue, I don't know how Vuforia could select OpenGLES2 as I specifically only allow Metal for rendering from Unity Really weird 😅

@zbagley
Copy link

zbagley commented Aug 7, 2019

The only Graphics API was OpenGLES2, when I add Metal it gives me il2cpp namespace errors:
image

If I only use OpenGLES2, the ARKit can't compile. From what I'm seeing the same trouble exists along EasyAR (from issue #123). Unless I'm missing something, it doesn't look like il2cpp supports ARKit/Metal. If that's the case, this project doesn't support ARKit.

I'll likely do a bit more exporation, but I'm already 40 hours deep into this and will likely try 2019.3 alpha with Unity as a library (#122) to see if it's more stable that this project and see what it would take to make the port.

@igrir
Copy link
Author

igrir commented Aug 7, 2019

@zbagley have you added -DNET_4_0 on your xcconfig? I tried to use ARKit utilizing Unity's AR Foundation and the camera seems working.

Unity as a library seems promising, and also being taken care of as it's official from Unity, it's superb. Maybe I'll do a little bit of research about it as well, but I don't know native development much. Hope we can integrate them using react-native as well later.

@zbagley
Copy link

zbagley commented Aug 7, 2019

@igrir Can't believe I forgot the flag (again). That was the issue, thank you.

@igrir
Copy link
Author

igrir commented Aug 7, 2019

update with my research. By the way I also post the questions on Vuforia forum, feel free to join in discussion (https://developer.vuforia.com/forum/unity/vuforia-always-set-opengl-es-20)

I was reading how VuforiaUnityPlayer.h works, and looks like there's a method called initQCARiOS. I tried to find all of the scripts containg 'initQCARiOS' and suspect there's a part called Vuforia.UnityExtensions1.cpp which had this line:

int32_t L_4 = IOSUnityPlayer_initQCARiOS_m84BC6F102A88721658DDAF7926D5F52922E4972D(L_1, L_2, L_3, /hidden argument/NULL);

I noticed on top of that line there's a method called GetRendererAPI, which maybe called Unity's choice of render API. I change the output to 2 (Metal) as explained in here: https://library.vuforia.com/content/vuforia-library/en/reference/unity/classVuforia_1_1VuforiaRenderer.html

The result it was initialized in Metal. Here's the dump code I gathered:

Initializing Vuforia...

2019-08-07 10:20:08.453624+0700 unityreact4[1208:149363] User pre-granted access to the camera

2019-08-07 10:20:08.453676+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Vuforia selected rendering API Metal

2019-08-07 10:20:08.453693+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Vuforia SDK version 8.3.8

2019-08-07 10:20:08.453707+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Vuforia SDK build 3621

2019-08-07 10:20:08.508179+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:08: Matched precache profile for Apple, iPad7,5, iOS 12, 800.1

2019-08-07 10:20:10.615716+0700 unityreact4[1208:149363] INFO/AR(1208) 2019-08-07 10:20:10: Vuforia Fusion: Detected and enabled use of ARKit

but the problem persists.

I also tried the delayed initialization method because maybe Vuforia being initialized before Unity component being called (as explained from this article https://medium.com/@xinyustudio/vuforia-delayed-initialization-explained-3eddfd02ac6), but the screen still black (but of course, the image recognition still working).

I investigate further and found this part of dump log:


2019-08-07 12:10:24.144059+0700 unityreact4[1260:187356] INFO/AR(1260) 2019-08-07 12:10:24: Vuforia Fusion: Detected and enabled use of ARKit

2019-08-07 12:10:24.144959+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView has CAMetalLayer layer class

2019-08-07 12:10:24.148389+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia

2019-08-07 12:10:24.148626+0700 unityreact4[1260:187356] DEBUG/AR(1260) Could not find a UIView with CAEAGLLayer or CAMetalLayer layer class that responds to selector renderFrameVuforia

2019-08-07 12:10:24.148846+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView has CAMetalLayer layer class

2019-08-07 12:10:24.149036+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia

2019-08-07 12:10:24.149225+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView has CAMetalLayer layer class

2019-08-07 12:10:24.149422+0700 unityreact4[1260:187356] DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia

2019-08-07 12:10:24.149628+0700 unityreact4[1260:187356] DEBUG/AR(1260) Could not find a UIView with CAEAGLLayer or CAMetalLayer layer class that responds to selector renderFrameVuforia

2019-08-07 12:10:24.151296+0700 unityreact4[1260:187356] INFO/AR(1260) 2019-08-07 12:10:24: Vuforia resumed.

the interesting part was "DEBUG/AR(1260) UIView does not respond to selector renderFrameVuforia"

@zbagley
Copy link

zbagley commented Aug 8, 2019

@igrir I must ask, did you run into any linking issues using RN 60.4? When linking I hit React/*.h missing type errors. When I create a custom podspec I run into UnityExport/*.h missing errors (added .podspec to node_modules library, imported react-native-unity-view in Podfile). Did you run into any of these sorts of issues, and how did you overcome them?

@igrir
Copy link
Author

igrir commented Aug 8, 2019

@zbagley Maybe this one? 👉 I did this which described by @ambewas on #122 (comment)

then in the library project itself, the header search paths need to be corrected, for React headers to be correctly resolved: add $(SRCROOT)/../../../ios/Pods/Headers/Public (or whatever relative path to that folder you've got) to library header search paths - set it to recursive.

@zbagley
Copy link

zbagley commented Aug 9, 2019

Verified 100x now that all the libraries are ported in, all caches are cleared in XCode, etc. etc. Any standard Unity project works, but I still don't believe this project is compatible with AR (in any form) for iOS.

Using the 2019.1 ARFoundation sample https://github.com/Unity-Technologies/arfoundation-samples/tree/2.1 for the Unity project, and the simple ar scene, the exported simply doesn't resolve the AR architecture properly.
image

Giving the number of problems we've encountered we will likely not continue to explore this repo's compatibility.

@jeserodz
Copy link

Hi @igrir. I'm experiencing the same issue you described. Have you been able to find a solution or workaround?

@Yeisonlop10
Copy link

Hi Good day. I see that this issue is still open. Is there any update on this?. I went through the same problems as you guys and our app works, but sometimes still crashes on startup, and I'm sure it is because of this. The app crashes when trying to execute UnityAppcontroller.mm - line 130 UnityInitApplicationGraphics(). Our app uses ARFoundations 1.5.0 preview 7, ARKit plugin 2.2.0 preview 6 in unity 2018.4.24.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants