Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

visionOS / Vision Pro support #203

Open
gwangyu-lee opened this issue Mar 21, 2024 · 12 comments
Open

visionOS / Vision Pro support #203

gwangyu-lee opened this issue Mar 21, 2024 · 12 comments
Assignees
Labels
enhancement New feature or request

Comments

@gwangyu-lee
Copy link

I'm having trouble building klakNDI in visionOS, but it works fine in iOS.

Unity Version: 2022.3.21f1

I got this error.
"CS0246: The type or namespace name 'Klak' could not be found”

@keijiro keijiro self-assigned this Mar 21, 2024
@keijiro keijiro added the question Further information is requested label Mar 21, 2024
@keijiro
Copy link
Owner

keijiro commented Mar 21, 2024

Could you try adding the visionOS platform to Klak.Ndi.Runtime.asmdef?

@gwangyu-lee
Copy link
Author

Thank you for the reply!

I changed it and I can build it now. But when I ran it on Xcode(using a simulator), I got these errors.

Could not locate file 'default-binaryarchive.metallib' in bundle.
Built from '2022.3/staging' branch, Version '2022.3.21f1 (bf09ca542b87)', Build type 'Development', Scripting Backend 'il2cpp'
MemoryManager: Using 'Dynamic Heap' Allocator.
[UnityMemory] Configuration Parameters - Can be set up in boot.config
    "memorysetup-allocator-temp-initial-block-size-main=262144"
    "memorysetup-allocator-temp-initial-block-size-worker=262144"
    "memorysetup-bucket-allocator-granularity=16"
    "memorysetup-bucket-allocator-bucket-count=8"
    "memorysetup-bucket-allocator-block-size=4194304"
    "memorysetup-bucket-allocator-block-count=1"
    "memorysetup-main-allocator-block-size=16777216"
    "memorysetup-thread-allocator-block-size=16777216"
    "memorysetup-gfx-main-allocator-block-size=16777216"
    "memorysetup-gfx-thread-allocator-block-size=16777216"
    "memorysetup-cache-allocator-block-size=4194304"
    "memorysetup-typetree-allocator-block-size=2097152"
    "memorysetup-profiler-bucket-allocator-granularity=16"
    "memorysetup-profiler-bucket-allocator-bucket-count=8"
    "memorysetup-profiler-bucket-allocator-block-size=4194304"
    "memorysetup-profiler-bucket-allocator-block-count=1"
    "memorysetup-profiler-allocator-block-size=16777216"
    "memorysetup-profiler-editor-allocator-block-size=1048576"
    "memorysetup-temp-allocator-size-main=4194304"
    "memorysetup-job-temp-allocator-block-size=2097152"
    "memorysetup-job-temp-allocator-block-size-background=1048576"
    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
    "memorysetup-temp-allocator-size-background-worker=32768"
    "memorysetup-temp-allocator-size-job-worker=262144"
    "memorysetup-temp-allocator-size-preload-manager=262144"
    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-gfx=262144"
-> applicationDidFinishLaunching()
Found 1 interfaces on host : 0) 192.168.0.21
Player connection [4371890752] Multi-casting "[IP] 192.168.0.21 [Port] 55000 [Flags] 2 [Guid] 186663699 [EditorId] 0 [Version] 1048832 [Id] VisionOSPlayer(50,Gwangyus-MacBook-Pro.local):56000 [Debug] 0 [PackageName] VisionOSPlayer [ProjectName] <no name>" to [225.0.0.222:54997]...

[Physics::Module] Initialized MultithreadedJobDispatcher with 15 workers.
[Subsystems] Discovering subsystems at path /Users/gwangyulee/Library/Developer/CoreSimulator/Devices/FC7AB30E-CB4B-4B7A-8CD3-19F680E2D2C0/data/Containers/Bundle/Application/4E084D14-5DF5-4E8A-9EC1-6EE4746F2C3E/KlakNDIURP.app/Data/UnitySubsystems
[Subsystems] No descriptors matched for  examples in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json.
[Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
[Subsystems] 1 'displays' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
[Subsystems] 1 'meshings' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
GfxDevice: creating device client; threaded=1; jobified=0
Initializing Metal device caps: Apple xrOS simulator GPU
Initialize engine version: 2022.3.21f1 (bf09ca542b87)
AddInstanceForFactory: No factory registered for id <CFUUID 0x600000230260> F8BB1C28-BAE8-11D6-9C31-00039315CD46
       MEMixerChannel.cpp:1006  MEMixerChannel::EnableProcessor: failed to open processor type 0x726f746d
[0x122164b80|InputElement #0|Initialize] Number of channels = 0 in AudioChannelLayout does not match number of channels = 2 in stream format.
The referenced script (Unknown) on this Behaviour is missing!
The referenced script on this Behaviour (Game Object '<null>') is missing!
WARNING: RGBA Compressed ASTC6X6 UNorm format is not supported, decompressing texture
XRGeneral Settings awakening...
UnityEngine.XR.Management.XRGeneralSettings:Awake()

[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Display...
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Input
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Input
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Meshing
[XR] VisionOSDisplayProvider::Initialize
[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Input...
[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Meshing...
New input system (experimental) initialized
[XR] Display Start
UnloadTime: 0.185666 ms
dyld[32781]: missing symbol called

But I can build and run it on devices and simulators.
Thank you for creating such a useful package!

@keijiro
Copy link
Owner

keijiro commented Mar 23, 2024

Thank you for confirming that. It's helpful since I don't have an actual device.

@gwangyu-lee
Copy link
Author

Here are a few updates.
I can build and run with real Vision Pro, using the iOS platform. Here is the screenshot.
IMG_0033

I can build but can't run with real Vision Pro, using the visionOS platform. Here is the log.
Screenshot 2024-03-24 at 5 11 25 AM

And I can build but can't run with a Vision Pro simulator, using the iOS and visionOS platforms.

@keijiro keijiro added enhancement New feature or request and removed question Further information is requested labels Jun 12, 2024
@keijiro keijiro changed the title Can't build to visionOS visionOS / Vision Pro support Jun 12, 2024
@gwangyu-lee
Copy link
Author

gwangyu-lee commented Aug 23, 2024

Hi! I fixed the error. There is no UnityEditor.visionOS.Xcode and libndi_visionos.a yet so I did some tricks.

  1. Make a visionOS folder in /Library/NDI SDK for Apple/lib/. Copy and paste libndi_ios.a into the folder. And change the name to libndi_visionos.a
Screenshot 2024-08-24 at 2 16 27 AM
  1. Use the UnityEditor.iOS.Xcode but edit the path manually.

PbxModifier.cs

// #if UNITY_IOS
using UnityEditor;
using UnityEditor.Callbacks;
using UnityEditor.iOS.Xcode;
using System.IO;

namespace Klak.Ndi.Editor
{

    public class PbxModifier
    {
        [PostProcessBuild]
        public static void OnPostprocessBuild(BuildTarget buildTarget, string path)
        {
            ModifyProjectFile(path);
            ModifyPlistFile(path);
        }

        static void ModifyProjectFile(string basePath)
        {
            // Set visionOS project file path
            var path = Path.Combine(basePath, "Unity-VisionOS.xcodeproj", "project.pbxproj");

            if (!File.Exists(path))
            {
                throw new FileNotFoundException($"Xcode project file not found at path: {path}");
            }

            var proj = new PBXProject();
            proj.ReadFromFile(path);

            var target = proj.GetUnityFrameworkTargetGuid();
            // Set visionOS lib file path
            var libPath = "/Library/NDI\\ SDK\\ for\\ Apple/lib/visionOS";
            proj.AddBuildProperty(target, "LIBRARY_SEARCH_PATHS", libPath);
            proj.AddFrameworkToProject(target, "Accelerate.framework", false);
            proj.AddFrameworkToProject(target, "VideoToolbox.framework", false);
            // Set the libndi_visionos.a file name
            proj.AddFrameworkToProject(target, "libndi_visionos.a", false);

            proj.WriteToFile(path);
        }

        static void ModifyPlistFile(string basePath)
        {
            var path = Path.Combine(basePath, "Info.plist");

            var plist = new PlistDocument();
            plist.ReadFromFile(path);

            var root = plist.root;

            // Bonjour service list
            {
                var key = "NSBonjourServices";
                if (root.values.ContainsKey(key))
                    root.values[key].AsArray().AddString("_ndi._tcp");
                else
                    root.CreateArray(key).AddString("_ndi._tcp");
            }

            // LAN usage description
            {
                var key = "NSLocalNetworkUsageDescription";
                var desc = "NDI requires device discovery capability " +
                           "on the networks you use.";
                if (!root.values.ContainsKey(key)) root.SetString(key, desc);
            }

            plist.WriteToFile(path);
        }
    }

} // namespace Klak.Ndi.Editor

// #endif

I just changed the project path like var path = Path.Combine(basePath, "Unity-VisionOS.xcodeproj", "project.pbxproj"); and libpath like var libPath = "/Library/NDI\\ SDK\\ for\\ Apple/lib/visionOS"; And this one. proj.AddFrameworkToProject(target, "libndi_visionos.a", false);

  1. Add visionOS platform to Klak.Ndi.Runtime.asmdef

Klak.Ndi.Runtime.asmdef

{
    "name": "Klak.Ndi.Runtime",
    "rootNamespace": "",
    "references": [
        "GUID:df380645f10b7bc4b97d4f5eb6303d95"
    ],
    "includePlatforms": [
        "Android",
        "Editor",
        "iOS",
        "visionOS",
        "LinuxStandalone64",
        "macOSStandalone",
        "WindowsStandalone64"
    ],
    "excludePlatforms": [],
    "allowUnsafeCode": true,
    "overrideReferences": false,
    "precompiledReferences": [],
    "autoReferenced": true,
    "defineConstraints": [],
    "versionDefines": [
        {
            "name": "com.unity.render-pipelines.core",
            "expression": "0.0.0",
            "define": "KLAK_NDI_HAS_SRP"
        }
    ],
    "noEngineReferences": false
}
  1. Now you can receive the NDI source but if you are using Mixed Reality app mode, the texture won't update. So you should make your texture dirty to update each frame. Please check this document PolySpatial Render Texture Support

So add this script and select your render texture.

SetRenderTextureDirty.cs

using UnityEngine;
using Unity.PolySpatial;

public class SetRenderTextureDirty : MonoBehaviour
{
    public RenderTexture texture;

    void Update()
    {
        Unity.PolySpatial.PolySpatialObjectUtils.MarkDirty(texture);
    }
}
Screenshot 2024-08-24 at 2 25 28 AM

This is the process I built and made the app.

p.s. I've made a Vision Pro app using this package. It is released on the Apple app store and you can download it for free until Aug 27. If you are interested, here is the link. Thank you.

@keijiro
Copy link
Owner

keijiro commented Aug 26, 2024

Thanks for giving the detailed information! I'll try it when the company device is available for testing.

@PRehker
Copy link

PRehker commented Sep 9, 2024

@gwangyu-lee Thanks for the experimentation and finding a solution! It is still failing for me, so I had a few quesitons:

  • For the 2nd step: do you mean KlakNDI/Editor/PbxModifier.cs, or is it some other file location?
  • Do you have to select your source at runtime, or will Unity remember the source you passed in the editor?
  • In XCode when building, the library (libndi_visionos.a) has been linked for iOS and it is giving me that warning. Did you relink the libndi library again for visionOS, or did it literally just work for you by changing the name?
    Thanks in advance! :)

@gwangyu-lee
Copy link
Author

Hi! @PRehker

  • For the 2nd step: Yes it is. Packages/KlakNDI/Editor/PbxModifier.cs
  • I cloned the KlakNDI repository and started the project in the URP folder. And edited all the scripts.
  • Do you mean this warning??
    ld: warning: building for platform=11, but linking in object file (/Library/NDI SDK for Apple/lib/visionOS/libndi_visionos.a(advertiser_bonjour.o)) built for iOS
    I just ignored this warning and NDI works for me.

Did you build your project with device SDK in Unity? I don't think NDI SDK works with iOS or visionOS simulator. And check that there is something in the project validation(Unity). Also, check that default command line tools are selected in Xcode - Settings - Locations

@gtk2k
Copy link

gtk2k commented Sep 10, 2024

@gwangyu-lee
I am also trying it with VisionOS, and I got a similar error in XCode.
So I am trying to create libndi_visionos.a, but there is no "NDI SDK for Apple".
Where can I get the "NDI SDK for Apple"?

@PRehker
Copy link

PRehker commented Sep 10, 2024

@gtk2k You have to install the NDI SDK from their website. You can get it here: https://ndi.video/for-developers/ndi-sdk/download/. After registering, you will recieve a mail with the download links. When you install the SDK, a new folder appears: "~/Library/NDI\ SDK\ for\ Apple/..."

@gtk2k
Copy link

gtk2k commented Sep 10, 2024

@PRehker
Thanks!

@gwangyu-lee
Copy link
Author

@PRehker
Here is the demo project you asked. Here is the link.
https://github.com/gwangyu-lee/KlakNDI-visionOS-Demo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants