Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PosterCalibration Camera Not Working in Unity 2018 #299

Open
jbienzms opened this issue Aug 14, 2018 · 13 comments
Open

PosterCalibration Camera Not Working in Unity 2018 #299

jbienzms opened this issue Aug 14, 2018 · 13 comments

Comments

@jbienzms
Copy link

I have been working on updating the PosterCalibration Sample to Unity 2018. I have everything compiling, I have all gestures working with the new APIs and I have voice commands working. However, I cannot get the camera feed to work and therefore posters are not detected.

My fork (and branch) for this work can be found here:

https://github.com/jbienzms/MixedRealityCompanionKit/tree/PosterUnity2018

The code deploys to the HoloLens and mostly runs, but when you say "Align Poster 1" the camera feed is just a white square. When connected to the Visual Studio debugger, the following is printed in the Output window:

AlignZone: CalibrationZone
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)


ImagePosterLocator:SetPosterTexture Mips:1
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)


ImagePosterLocator:SetPosterTexture - using mip #0:1024x622
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)


ImagePosterLocator:StartProcessing: CalibrationZone
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)


Update version updated.
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)


PhotoCapture created...
 
(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 43)


Photo Mode has already been started.
 
(Filename: C:\buildslave\unity\build\Runtime/VR/HoloLens/WebCam/PhotoCapture.cpp Line: 530)


Exception thrown: 'System.NotImplementedException' in WinRTBridge.dll
'PosterCalibrationSample.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\PosterCalibrationSampleVS.Release_x86.jbienz\System.Diagnostics.StackTrace.dll'. Cannot find or open the PDB file.
Exception: The method or operation is not implemented.
Type: System.NotImplementedException
Module: WinRTBridge
InnerException: <No Data>
AdditionalInfo:MarshalFirstFieldIntoNativePtr type: UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame
   at WinRTBridge.Marshalling.MarshalFirstFieldIntoNativePtr(Object instance)
 
(Filename:  Line: 0)


Exception thrown: 'System.ArgumentNullException' in UnityEngineProxy.dll
ArgumentNullException: Value cannot be null.
Parameter name: _unity_self
   at UnityEngineProxy.InternalCalls.PhotoCaptureFrame_CUSTOM_GetDataLength(Object self)
   at UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame..ctor(IntPtr nativePtr)
   at UnityEngine.XR.WSA.WebCam.PhotoCapture.InvokeOnCapturedPhotoToMemoryDelegate(OnCapturedToMemoryCallback callback, Int64 hResult, IntPtr photoCaptureFramePtr)
   at UnityEngine.XR.WSA.WebCam.PhotoCapture.$Invoke10(Int64 instance, Int64* args)
   at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64* args, IntPtr method) 
(Filename: <Unknown> Line: 0)
@jbienzms
Copy link
Author

@dberrett Hey Doug. Any thoughts on this? Is this happening in the unmanaged plugin? The plugin still compiles, but it appears to not be getting the expected pointer back, and a NotImplementedException seems rather unexpected.

@dberrett
Copy link
Member

dberrett commented Aug 14, 2018 via email

@jbienzms
Copy link
Author

Forgive my ignorance on this one, but why is the C++ plugin project compiling correctly if the method signature changed?

@dberrett
Copy link
Member

dberrett commented Aug 15, 2018 via email

@jbienzms
Copy link
Author

Actually, I was pretty sure this was in the C++ code. I could be wrong, but the first exception is the System.NotImplementedException which at least appears to have occurred in PosterCalibrationSampleVS.Release_x86.

AdditionalInfo:MarshalFirstFieldIntoNativePtr type: UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame

I could be wrong about that, but that's the info we're getting in the output window.

@dberrett
Copy link
Member

dberrett commented Aug 15, 2018 via email

@jbienzms
Copy link
Author

No, you're right @dberrett . For some reason I was thinking the VS project was the C++ project but you're right that it's the Unity project exported to VS. Is this a new Unity bug that we need to open with them?

Before we decide that for sure, I just want to make sure we're not doing something we shouldn't be. I noticed this message as well:

Photo Mode has already been started.

Perhaps there is a bug trying to initialize the camera twice?

@dberrett
Copy link
Member

dberrett commented Aug 15, 2018 via email

@MSAlshair
Copy link

MSAlshair commented Oct 12, 2018 •
edited

@jbienzms : I think this is a Unity problem. I have been trying to resolve this for few days. Last successful line of code is photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory). But it actually never enters the OnCapturedPhotoToMemory method. After figuring out where it is failing, I copied unity sample exactly as shown in their website https://docs.unity3d.com/Manual/windowsholographic-photocapture.html and it still fails.
This happened when I tried to upgrade my project to Unity 2018.2. I tried to set it to IL2CPP and it compiled correctly, but it failed displaying results when I deployed it to the device. I assumed it was azure initially, but I was not able to debug it using C++. I tried to attach to debug installed app package to be able to debug in C#, I connected successfully, but failed to hit any break points and debugging this issue using C++ was painful for me. Finally, after so many days, I decided to switch it back to .NET backend and debugged the project to come to the conclusion that it is a unity problem. Tomorrow, I will try the beta version and keep trying prior versions to determine when is the last working version.
Side note:
Do you know how to debug the C# project when you use IL2CPP backend? I know how make it compile correctly, but not debug it. It fails to attach unity debugger to the hololens device. But visual studio allows me to attach debugger to the deployed application, but it will not hit the C# code, I chose managed and then i switched to native, then switched to both and none worked. According to this video from unity, it should work:
https://oc.unity3d.com/index.php/s/rx7KD0SYeQXr6qn
and this is the unity thread that I got the video from
https://oc.unity3d.com/index.php/s/rx7KD0SYeQXr6qn

I will update you when I figure out something.

@Troy-Ferrell
Copy link
Contributor

Hi all,

I was able to get the project working in Unity 2018.1.9f2 and HoloLens April 2018.

I did the following to get it working:

  • Update the obsolete interaction APIs (i.e from "SourcePressed" to "InteractionSourcePressed").
  • Ensure ImagePosterLocator component on CalibrationZone object is hooked up to PVCamView & PosterDetectView gameobject appropriately
  • AutoAlignToPoster has Poster gameobject set correctly
  • SpeechCommands component/gameobject is connected to the CalibrationZone (ZoneCalibrationManager)
  • Only build the single-zone scene

I have the configuration for building as follows (under PlayerSettings/OtherSettings):
Scripting Runtime Version: .Net 3.5 equivalent
Scripting Backend: .NET
API Compatibility Level: .Net 4.x

Further, make sure you have capabilities enabled for WebCam and Microphone under PlayerSettings/Publishing Settings

Try to match to this parity.

Thanks,
Troy

@MSAlshair
Copy link

MSAlshair commented Oct 15, 2018 •
edited

@Troy-Ferrell Thank you for your help. My issue is a little different, I was trying to use the camera directly to take photos. I apologize for posting my issue here. I saw that @jbienzms has the same camera problem even though we are doing two different things.

  1. I created a workaround for the camera using MediaCapture:
    https://github.com/MSAlshair/HoloLensMediaCapture
  2. FYI: There is a Unity public active tracker:
    https://issuetracker.unity3d.com/issues/windowsmr-failure-to-take-photo-capture-in-hololens
    Thank you again for your help!

@jbienzms
Copy link
Author

Thank you so much @MSAlshair for your workaround and taking the time to publish a sample. By any chance have you submitted this bug to Unity? And if not, any chance you could submit it? We'd like to know the bug number so we can start tracking it.

@MSAlshair
Copy link

MSAlshair commented Oct 22, 2018 •
edited

@jbienzms : You are very welcome. I didn't create a bug with Unity, but as I stated in my last post, I found an active bug that was already submitted since April 2018 and it is still open

Unity Bug:

https://issuetracker.unity3d.com/issues/windowsmr-failure-to-take-photo-capture-in-hololens

Unity forms discussing same issue:

https://forum.unity.com/threads/hololens-photo-capturing-failing.548845/

https://forum.unity.com/threads/photocapture-not-called-while-running-in-hololens.541825/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@MSAlshair @jbienzms @dberrett @Troy-Ferrell