Unity3D plugin

[zaherj]

Hi,

Do you have plan to release a plugin for Unity3D?

[thundercarrot]

Yes, but I can't say when.

You might be interested in the 'desktopDuplication' option in the projection mapping sample (it is described briefly in the README). It's a quick way to get the projection mapping sample to work with any program that renders to the screen. The support for it in the sample code is a bit sketchy-- some things are hard coded, but depending on your needs, it may be a very nice solution. Also note that in CalibrateEnsemble you can save the calibrated room geometry to an .OBJ file that is easily imported to programs like Unity.

[joshferns]

Hi,
After calibration, is it possible to use the transformation matrix found in calibration.xml and apply it in unity(bypassing roomalive after calibration).
Or are things more complicated than that?

[thundercarrot]

Yes. One thing you have to deal with is the handedness of the coordinate systems. RoomAliive Toolkit uses a right handed coordinate system (like Kinect), while Unity uses a left handed coordinate system.

[joshferns]

Thanks,
How do I get the matrix that will get the output rendered from the Kinect's point of view rather than the projectors point of view?

[thundercarrot]

Look at how GraphicsTransforms.ProjectionMatrixFromCameraMatrix is called in CalibrateEnsemble.or the sample. You can create a projection matrix for the Kinect depth or color camera in the same way that the sample creates a projection matrix for the projector. You'll need a view matrix for the color or depth camera as well. You can see how this is done in the sample for the projector; in you need the view matrix that takes a point in the world coordinate frame into the camera you are interested in.

[hopexiao]

Has anybody integrated with Unity successfully?

[zaherj]

Hi Guys

I am having trouble getting the projection matrix correct in unity3d.

I tried to use the same method that GraphicsTransforms.ProjectionMatrixFromCameraMatrix uses.

However GraphicsTransforms.ProjectionMatrixFromCameraMatrix calculates matrix that projects to a volume where x,y in [-1,1] and in z[0,1]

Is that because SharpDX assumes that the viewport clipping volume ranges from -1.0 to 1.0 in X, and from 1.0 to -1.0 in Y?

Unity3D assumes that the viewport clipping volume ranges from 0 to 1 for both X and Y (x right, y up) so how do I calculate the projection matrix so that it projects to a volume where x,y in [0,1]?

Thanks

[zaherj]

I finally was able to get the projection matrix correct in unity3d :)

So I would like to correct my previous post and provide a code snippet to how you can calculate the projection matrix for Unity3D 👍

So one thing I learned (please correct if I am wrong) is that Canonical View Volume and ViewPort are two different things: Canonical View Volume is 3d Box where the projection matrix projects into, and ViewPort is 2d rectangle that takes the X,Y values of Canonical View Volume and normalize them to some defined range which is (0,1) for both X and Y (X is right, Y is up) in Unity3D. So projection matrix has nothing to do with ViewPort.

The reason that GraphicsTransforms.ProjectionMatrixFromCameraMatrix didn't work out of the box is that RoomAlive uses RH coordinate system as @thundercarrot mentioned, and Unity3D uses LH coordinate system, so we need to tweak the matrix to match the correct coordinate system.

Here is what I did

float dx = cx - w / 2;
float dy = cy - h / 2;

float near = 0.1f;
float far = 100.0f;


Matrix4x4 projectionMatrix  = new Matrix4x4();

Vector4 row0 = new Vector4((2f * fx / w), 0,(2f * dx / w),0);
Vector4 row1 = new Vector4(0,2f * fy / h,-2f * (dy +1f) / h,  0);
Vector4 row2 = new Vector4(0,0,-(far +near) / (far - near),-near *(1+ (far+near)/(far-near)));
Vector4 row3 = new Vector4(0,0,-1, 0);

projectionMatrix.SetRow(0, row0);
projectionMatrix.SetRow(1, row1);
projectionMatrix.SetRow(2, row2);
projectionMatrix.SetRow(3, row3);

Now my projectors see what they project :)

Thanks

[joshferns]

Great to hear that @zaherj ,
Will hopefully try to test out what you've been able to achieve.
Could you possibly share a sample video?

[hbenko]

Since we have now released a full Unity codebase, this issue is closed.