Please help: DX11(D3DX) matrix & clipping plane problems

[xjkkoski]

Hi,

I'm trying to implement VR for my old driving game. I used "directx11 hellovr" (or such) as a base, and "converted" the matrix related code to D3DX (which my game use). But the camera is facing backwards in the HMD for one, so something is not quite right, unfortunately I don't understand much about matrix calculations. And then there seems to be some near/far clipping plane problem, I'm able to change the near/far values realtime, but can't come up any values to properly fix it.

Anyway, here is a video on how it looks, hopefully it will help to understand the problem.
I move my head around in the HMD, but it doesn't even look like the view is really properly turning as it should be. Plus all sorts of other problems. At 0:45 I briefly turn the VR off so you can see how the surroundings should really look like.

https://youtu.be/WzIY-lTL4Xw

I will add some related code in the next post. Let me know if you need something more.

Thanks!

[xjkkoski]

// At start of rendering
if (eye==0) {
	steamVR.UpdateHMDMatrixPose();
	g.projectionMatrix=steamVR.getCurrentViewProjectionMatrix(vr::Eye_Left);
}
if (eye==1) g.projectionMatrix=steamVR.getCurrentViewProjectionMatrix(vr::Eye_Right);

...

void SteamVR::setupCameras()
{
	m_mat4ProjectionLeft = GetHMDMatrixProjectionEye(vr::Eye_Left);
	m_mat4ProjectionRight = GetHMDMatrixProjectionEye(vr::Eye_Right);
	m_mat4eyePosLeft = GetHMDMatrixPoseEye(vr::Eye_Left);
	m_mat4eyePosRight = GetHMDMatrixPoseEye(vr::Eye_Right);
}

D3DXMATRIX SteamVR::getCurrentViewProjectionMatrix(vr::Hmd_Eye nEye)
{
	D3DXMATRIX matMVP;
	if (nEye == vr::Eye_Left)
	{
		matMVP = m_mat4ProjectionLeft * m_mat4eyePosLeft * m_mat4HMDPose;
	}
	else if (nEye == vr::Eye_Right)
	{
		matMVP = m_mat4ProjectionRight * m_mat4eyePosRight *  m_mat4HMDPose;
	}

	return matMVP;
}

D3DXMATRIX SteamVR::GetHMDMatrixProjectionEye(vr::Hmd_Eye nEye)
{
	if (!m_pHMD)
		return D3DXMATRIX();
	
	vr::HmdMatrix44_t mat = m_pHMD->GetProjectionMatrix(nEye, g.nearPlane, g.farPlane); // values are g.nearPlane=1.0, g.farPlane=35000.0

	return D3DXMATRIX(
		mat.m[0][0], mat.m[1][0], mat.m[2][0], mat.m[3][0],
		mat.m[0][1], mat.m[1][1], mat.m[2][1], mat.m[3][1],
		mat.m[0][2], mat.m[1][2], mat.m[2][2], mat.m[3][2],
		mat.m[0][3], mat.m[1][3], mat.m[2][3], mat.m[3][3]
		);
}

D3DXMATRIX SteamVR::GetHMDMatrixPoseEye(vr::Hmd_Eye nEye)
{
	if (!m_pHMD)
		return D3DXMATRIX();

	vr::HmdMatrix34_t matEyeRight = m_pHMD->GetEyeToHeadTransform(nEye);
	D3DXMATRIX matrixObj(
		matEyeRight.m[0][0], matEyeRight.m[1][0], matEyeRight.m[2][0], 0.0,
		matEyeRight.m[0][1], matEyeRight.m[1][1], matEyeRight.m[2][1], 0.0,
		matEyeRight.m[0][2], matEyeRight.m[1][2], matEyeRight.m[2][2], 0.0,
		matEyeRight.m[0][3], matEyeRight.m[1][3], matEyeRight.m[2][3], 1.0f
		);

	D3DXMATRIX matTemp;
	D3DXMatrixInverse(&matTemp, NULL, &matrixObj); // todo: just use the matrixObj for pOut
	return matTemp; 

}

D3DXMATRIX SteamVR::ConvertSteamVRMatrixToMatrix4( const vr::HmdMatrix34_t &matPose )
{
	D3DXMATRIX matrixObj(
		matPose.m[0][0], matPose.m[1][0], matPose.m[2][0], 0.0,
		matPose.m[0][1], matPose.m[1][1], matPose.m[2][1], 0.0,
		matPose.m[0][2], matPose.m[1][2], matPose.m[2][2], 0.0,
		matPose.m[0][3], matPose.m[1][3], matPose.m[2][3], 1.0f
		);
	return matrixObj;
}

void SteamVR::UpdateHMDMatrixPose()
{
	if (!m_pHMD)
		return;
	vr::VRCompositor()->WaitGetPoses(m_rTrackedDevicePose, vr::k_unMaxTrackedDeviceCount, NULL, 0);

	m_iValidPoseCount = 0;
	m_strPoseClasses = "";
	for (int nDevice = 0; nDevice < vr::k_unMaxTrackedDeviceCount; ++nDevice)
	{
		if (m_rTrackedDevicePose[nDevice].bPoseIsValid)
		{
			m_iValidPoseCount++;
			m_rmat4DevicePose[nDevice] = ConvertSteamVRMatrixToMatrix4(m_rTrackedDevicePose[nDevice].mDeviceToAbsoluteTracking);
			if (m_rDevClassChar[nDevice] == 0)
			{
				switch (m_pHMD->GetTrackedDeviceClass(nDevice))
				{
				case vr::TrackedDeviceClass_Controller:        m_rDevClassChar[nDevice] = 'C'; break;
				case vr::TrackedDeviceClass_HMD:               m_rDevClassChar[nDevice] = 'H'; break;
				case vr::TrackedDeviceClass_Invalid:           m_rDevClassChar[nDevice] = 'I'; break;
				case vr::TrackedDeviceClass_GenericTracker:    m_rDevClassChar[nDevice] = 'O'; break;
				case vr::TrackedDeviceClass_TrackingReference: m_rDevClassChar[nDevice] = 'T'; break;
				default:                                       m_rDevClassChar[nDevice] = '?'; break;
				}
			}
			m_strPoseClasses += m_rDevClassChar[nDevice];
		}
	}

	if (m_rTrackedDevicePose[vr::k_unTrackedDeviceIndex_Hmd].bPoseIsValid)
	{
		//m_mat4HMDPose = m_rmat4DevicePose[vr::k_unTrackedDeviceIndex_Hmd].invert();
		D3DXMatrixInverse(&m_mat4HMDPose, NULL, &m_rmat4DevicePose[vr::k_unTrackedDeviceIndex_Hmd]);
		//cprint("[SteamVR] pose updated");
	}
	else
	{
		//cprint("[SteamVR] pose not valid");
	}
}

[Rectus]

Not sure if this is the issue, but the documentation for D3DXMATRIX says that the _34 element of a projection matrix cannot be a negative number. SteamVR seems to output matrices that have that element negative. Try multiplying the matrix in GetHMDMatrixProjectionEye() with -1.

[xjkkoski]

Thanks for the reply! I tried the multiplication but could not get it to work so for now I restored it.

As far as I know the matrix format should basicly be the same in the dxHelloWorld sample I derived my code from. So, I set the HMD to fixed position and tried to compare the matrix prints from the sample application and my game (using same near/far settings). The values look pretty identical except the last row of the final matrix, in my game it comes just as 0.00000, 0.00000, 0.00000, 1.00000, while in the sample it's some actual values. I'm not even going try to understand what I'm seeing, it doesn't make sense to me, so I will just ask, what could cause the difference in my game and the sample app?

Here is the data as text:
[dxHelloworld] pleft =
0.99939, 0.00000, 0.03490, 0.00000,
0.00000, 0.93007, -0.06993, 0.00000,
0.00000, 0.00000, -1.00334, -0.10033,
0.00000, 0.00000, -1.00000, 0.00000,
***[disle] 32.223: m_mat4ProjectionLeft:
0.99939, 0.00000, 0.03490, 0.00000,
0.00000, 0.93007, -0.06993, 0.00000,
0.00000, 0.00000, -1.00334, -0.10033,
0.00000, 0.00000, -1.00000, 0.00000,

[dxHelloworld] , eyeLeft =
1.00000, 0.00000, 0.00000, 0.03175,
0.00000, 1.00000, 0.00000, -0.00000,
0.00000, 0.00000, 1.00000, -0.00000,
0.00000, 0.00000, 0.00000, 1.00000,
***[disle] m_mat4eyePosLeft:
1.00000, 0.00000, 0.00000, 0.03175,
0.00000, 1.00000, 0.00000, 0.00000,
0.00000, 0.00000, 1.00000, 0.00000,
0.00000, 0.00000, 0.00000, 1.00000,

[dxHelloworld] , hmdPose =
0.40232, 0.00361, -0.91549, 0.18208,
0.03069, 0.99938, 0.01743, -0.77019,
0.91499, -0.03511, 0.40196, 0.73907,
0.00000, 0.00000, 0.00000, 1.00000,
***[disle] 32.223: m_mat4HMDPose:
0.40142, 0.00334, -0.91589, 0.18453,
0.03071, 0.99938, 0.01711, -0.76916,
0.91538, -0.03499, 0.40106, 0.73970,
0.00000, 0.00000, 0.00000, 1.00000,

[dxHelloworld] final =
0.43401, 0.00239, -0.90091, 0.23950,
-0.03544, 0.93195, -0.01190, -0.76802,
-0.91805, 0.03522, -0.40330, -0.84188,
-0.91499, 0.03511, -0.40196, -0.73907,
***[disle] 32.224: final:
0.40117, 0.00311, 0.73545, 0.09190,
0.03069, 0.92950, 0.68221, -0.00172,
0.91482, -0.03255, -1.13678, -0.04024,
0.00000, 0.00000, -1.00000, 0.00000,

Edit: Looks like there's also other differences in the final matrix, not just the last row..

[Rectus]

Looks like a D3DXMATRIX is stored in a row major format, while the Matrix4 class in the example is stored in a column major format. The example transposes the matrices when converting them to convert between the different formats, so you would need to change the functions to copy them without the transpose like this:

return D3DXMATRIX(
		mat.m[0][0], mat.m[0][1], mat.m[0][2], mat.m[0][3],
		mat.m[1][0], mat.m[1][1], mat.m[1][2], mat.m[1][3],
		mat.m[2][0], mat.m[2][1], mat.m[2][2], mat.m[2][3],
		mat.m[3][0], mat.m[3][1], mat.m[3][2], mat.m[3][3]
		);

If that doesn't work, it might also be necessary to reverse the multiplication order of the MVP matrix as well.

[xjkkoski]

Thank you for taking the time to help!

I changed the matrices like you suggested and reversed the multiplication. Initially there was nothing recgonizable on the screen, but once I removed the D3DXMatrixInverse calls from GetHMDMatrixPoseEye and UpdateHMDMatrixPose things started working better. Camera was still facing backwards though, but now the view turns properly with the HMD!

With the following "trick" (something I've found searching here at some point I think) I can get the camera to face forward, and it works pretty good, just some annoying blue "artifact" moving on the screen.

Here's an quick video, I enable the UpdateHMDMatrixPose D3DXMatrixInverse call from the console to show it's effect, in GetHMDMatrixPoseEye the call just causes nothing to be visible on screen. Finally I enable the "trick" above.

https://youtu.be/Rth4TcCrWGs

Edit: I got rid of the blue artifact with just adjusting the near plane value, but having the HMD actually on my head I noticed that when I move my head forward, steering wheel does not come any closer but just kinda follows my head, same thing pulling back my head, the distance to steering wheel just keeps more or less the same. I'm guessing this has to do with the "trick", which isn't doing the trick quite properly.

I'll fiddle more with it after work. Anyway thanks again!

[xjkkoski]

I ended up using the original Matrix4 code exactly as is in the original example, and at the very end (getCurrentViewProjectionMatrix) I convert it to D3DXMATRIX. The camera still does not follow the head movement backward/forward, but I think this is good enough for my application. Looking around is now atleast natural, there was something a bit off in it earlier. Anyway thanks again, now I just have to solve some other issues but those are, well, other issues.