Support for VR Android Build

[keli95566]

Hi there! Thank you very much for making this great project open source! I would very much like to make it work on Android, specifically on standalone virtual reality headsets like vive focus3 and oculus Quest2.
It seems that Vive focus 3 supports arm v7, arm 64 is preferred on Quest2. And I would like to test the basic armv7 avaliable first. I tried to use the Android branch and built to Vive Focus3, the images are not streaming. The Android branch also doesn't work if I just run on Windows Unity Editor, such as the testScene1 or test scene2. The editor output a dllnotfound error:

DllNotFoundException: GStreamerUnityPlugin
GStreamerCore.Ref () (at Assets/UnityGstream/GStreamerUnity/Scripts/GStreamerCore.cs:130)
GstBaseTexture.Initialize () (at Assets/UnityGstream/GStreamerUnity/Scripts/GstBaseTexture.cs:88)
CustomPipelinePlayer.Start () (at Assets/UnityGstream/GStreamerUnity/Components/CustomPipelinePlayer.cs:26)

But the pre-built dll files I pulled directly from the repo are still there.

And if I tried to re-build the plugin myself, I got the following errors from Visual Studio 2019. I am not sure what they mean and how I could solve the bugs..

I am not sure how I could proceed from here. It would really be awesome if there are still some sort of supports to continue this project! As streaming through UDP is a very important technology for VR applications, Gstreamer works really fast across the network. This is the only open source project that currently support Unity 2020 LTS + ( The master branch works flawlessly in Editor), which most of VR environments will run on! Thank you so much in advance for anyone who could provide guidance!

[keli95566]

Update:
I logged the status in VR and it shows that network and connection to gstreamer is successful.

And I also rebuild the Android plugin for the renderAPI to make sure it imports OpenGLES3 for VR.

But still, nothing is showing up from the video stream, not actual texture from video stream is rendered.....

[jbest2015]

Hello was this ever resolved, I am headed in the same direction

[roguestarslade]

@jbest2015 @keli95566 @mrayy I think I know what's going on here.

From my day job, we've had some issues with Android GStreamer outside of version 1.14.4, we attempted to actually use GStreamer 1.16.x and also 1.18.x, which produced a lot of errors.

Also, Oculus VR requires an ArmV8a-64 build, which is not being made by the make files in this project.

I'm going to attempt making some tweaks to the make files and allow for arm 64 builds, lemme see if that works.

[JamesDeCarlo]

If you do get it working with 1.16.x arm64 would be very interested myself. I started trying to get it working myself but had to abandon for higher priorities. Good luck on the pursuit and please keep us posted!

[roguestarslade]

@JamesDeCarlo I doubt 1.16.x will work.

Most likely I will need to revert this to 1.14.4 to get this repo working universally with Windows + Android

[palwanishan]

Greetings,
@roguestarslade
did you get it working? Does anyone have any progress?

[keli95566]

Hi there!
Yes, it ended up working well!
Downgrading to Gstreamer 1.14.5 works.
You also need to rebuild the android branch of this repo for arm64 architecture using the NDK toolkit. This will result in three plugin files: libGStreamerUnityPlugin.so, libRenderUnityPlugin.so, libgstreamer_android.so

You can download the built plugin files here: https://github.com/keli95566/MRTunnelingPico/tree/main/MRTunnelingPico/Assets/Plugins/Android

I used this package to build a streamer for the ZED Mini camera for stereoscopic VR applications. Feel free to take a look at the repo.

[sealfoss]

@keli95566

Hi Keli, were you able to get your compiled plugin binaries running on Oculus Quest, specifically? I've added your .so files to my plugin directory, and I've deployed the application to my Oculus Quest 2. However, at startup, the application hangs and eventually dies. Here is the end of the output from logcat -s Unity:

02-10 11:03:49.446 27334 27358 D Unity   : RenderTexture::Create (id: 5)
02-10 11:03:49.446 27334 27387 D Unity   : IVRDeviceCallback_CreateEyeTextureResources 0x1a
02-10 11:03:49.446 27334 27358 D Unity   : SetupRenderTextureFromXRRequest (id: 6 col: 0x1d d: 0x0 sr: 0x0)
02-10 11:03:49.446 27334 27358 D Unity   : RenderTexture::Create (id: 6)
02-10 11:03:49.446 27334 27387 D Unity   : IVRDeviceCallback_CreateEyeTextureResources 0x1d
02-10 11:03:53.862 27334 27358 D Unity   : UnloadTime: 0.948281 ms
02-10 11:03:53.892 27334 27358 I Unity   : initializing gstreamer
02-10 11:03:53.892 27334 27358 I Unity   : GStreamerCore:Ref()
02-10 11:03:53.892 27334 27358 I Unity   : BaseVideoPlayer:Start()
02-10 11:03:53.892 27334 27358 I Unity   :

Do you have any guidance here? Did you run into any similar problems? Can I ask what version of the NDK was used to build your binaries?

[BerkDevanthro]

@keli95566

Hi Keli, were you able to get your compiled plugin binaries running on Oculus Quest, specifically? I've added your .so files to my plugin directory, and I've deployed the application to my Oculus Quest 2. However, at startup, the application hangs and eventually dies. Here is the end of the output from logcat -s Unity:

02-10 11:03:49.446 27334 27358 D Unity   : RenderTexture::Create (id: 5)
02-10 11:03:49.446 27334 27387 D Unity   : IVRDeviceCallback_CreateEyeTextureResources 0x1a
02-10 11:03:49.446 27334 27358 D Unity   : SetupRenderTextureFromXRRequest (id: 6 col: 0x1d d: 0x0 sr: 0x0)
02-10 11:03:49.446 27334 27358 D Unity   : RenderTexture::Create (id: 6)
02-10 11:03:49.446 27334 27387 D Unity   : IVRDeviceCallback_CreateEyeTextureResources 0x1d
02-10 11:03:53.862 27334 27358 D Unity   : UnloadTime: 0.948281 ms
02-10 11:03:53.892 27334 27358 I Unity   : initializing gstreamer
02-10 11:03:53.892 27334 27358 I Unity   : GStreamerCore:Ref()
02-10 11:03:53.892 27334 27358 I Unity   : BaseVideoPlayer:Start()
02-10 11:03:53.892 27334 27358 I Unity   :

Do you have any guidance here? Did you run into any similar problems? Can I ask what version of the NDK was used to build your binaries?

Did you find any solution for this? I'm trying to run it on Quest Pro but it crashes at the same lines (as soon at it uses the DLL references).