New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting the projection matrix of camera #14

Open
dineshreddy91 opened this Issue Sep 20, 2016 · 17 comments

Comments

Projects
None yet
8 participants
@dineshreddy91
Copy link

dineshreddy91 commented Sep 20, 2016

I would like to get the projective camera matrix of the rendering engine. But the code is commented out in the plugin.

FExecStatus FCameraCommandHandler::GetCameraProjMatrix(const TArray& Args)
{
// FMatrix& ProjMatrix = FSceneView::ViewProjectionMatrix;
// this->Character->GetWorld()->GetGameViewport()->Viewport->
// this->Character
return FExecStatus::InvalidArgument;
}

Is it possible to add this feature??

If the camera matrix is fixed can you send me the matrix information for FOV 90.

@qiuwch

This comment has been minimized.

Copy link
Member

qiuwch commented Sep 20, 2016

Hi, @dineshreddy91

This is an unfinished feature and I am working on it.
The camera matrix for FOV 90 is fixed, I need to ask my co-author for the exact value.

@dineshreddy91

This comment has been minimized.

Copy link

dineshreddy91 commented Sep 21, 2016

@qiuwch thanks for the reply. can you provide the projection matrix for FOV 90. i wanted to test the odometry and am stuck without the intrinsic matrix of the camera.

@edz-o

This comment has been minimized.

Copy link
Member

edz-o commented Sep 21, 2016

Hi, @dineshreddy91

The focal length f = W / 2 / tan(FOV/2) = W/2, where W is the width of the image which is 640 px here. And the pixels are square.

@dineshreddy91

This comment has been minimized.

Copy link

dineshreddy91 commented Sep 21, 2016

Hi @edz-o thanks for the reply
According to your formula the k matrix comes out to be

K = [320 0 320
0 320 240
0 0 1]
The depth to point cloud conversion using this intrinsic matrix is very distorted. Previously for opengl to camera matrix conversion i have used this https://strawlab.org/2011/11/05/augmented-reality-with-OpenGL. i wanted to access the OpenGL projection matrix to convert to the Hz intrinsic matrix.

@qiuwch

This comment has been minimized.

Copy link
Member

qiuwch commented Sep 22, 2016

@dineshreddy91 If this is not what you are looking for, then I need to spend one or two days to read Unreal Engine's code to find what is the correct answer.

@qiuwch qiuwch added the enhancement label Sep 25, 2016

@qiuwch qiuwch modified the milestone: v0.3.0 Oct 17, 2016

@debadeepta

This comment has been minimized.

Copy link

debadeepta commented Jun 8, 2017

screenshot from 2017-06-07 17-09-52
Using the K matrix above I am getting a really distorted point cloud as well. See attached image. It is a view into the room in RealisticRendering binary example. The depth image converted to point cloud using K above makes the scene 'cave-in' towards the center. I think we need the camera distortion model and coefficients to properly project. Is anyone else having the same experience?
figure_1

@edz-o

This comment has been minimized.

Copy link
Member

edz-o commented Jun 8, 2017

Hi,

@debadeepta
We guess the cause is that the depth image obtained by vget /camera/0/depth is the distance from the scene to the camera center, you may convert it to the distance to image plane using this function.

@debadeepta

This comment has been minimized.

Copy link

debadeepta commented Jun 9, 2017

Thank you! I didn't realize the depth image was in ray-distance form. Your function works great. Just a small nitpick: 'f' is undefined in your function. Ideally should be an argument to it.

Many thanks!

@bhack

This comment has been minimized.

Copy link

bhack commented Jun 10, 2017

Could we add this to the plugin?

@qiuwch

This comment has been minimized.

Copy link
Member

qiuwch commented Jun 10, 2017

@bhack I am willing to finish the uncompleted vget /camera/[id]/proj_matrix. I am working on #60 and it is almost finished. This will be my next step. If anyone has already done this and can make a pull request, that would be best.

@debadeepta I think the distance to the image plane is a much natural definition for depth. The ray-distance should at least be clearly documented, this is our fault. I am thinking to provide both these two types of depth, but not sure how to call them, depth and ray_distance?

@DonnEssime

This comment has been minimized.

Copy link

DonnEssime commented Jun 16, 2017

I have a quick note: the posted distance-to-image plane function does not seem to be correct; it actually computes the distance to the camera along the viewing direction (i.e. distance to image plane + focal depth).
The correct equation on line 8 should, imo, read

PlaneDepth = (PointDepth-1) / (1 + (DistanceFromCenter / f)**2)**(0.5)

However, for single-image reconstruction purposes it seems to work as it should 👍 I guess this is more of a semantic note, in that it does not compute the distance to the image plane, but rather to the Z plane in the camera's reference system.

@qiuwch

This comment has been minimized.

Copy link
Member

qiuwch commented Jun 18, 2017

Hi, @DonnEssime

@edz-o is travelling these days, I will ask him to follow up this issue when he is back.

@edz-o

This comment has been minimized.

Copy link
Member

edz-o commented Jun 21, 2017

Hi, @DonnEssime

You are right, this function computes the distance to the camera plane, i.e. the Z plane.

@DonnEssime

This comment has been minimized.

Copy link

DonnEssime commented Jun 21, 2017

For future reference, I got confused with the calculation/application of the location and rotation, because they do not follow the axes lay-out I was expecting. The translation axes for a non-rotated camera are given with the x axis into the scene, the y axis to the right and the z axis to the top (left-handed, for some reason?). I was expecting it to be z axis into the scene, x axis to the right and y axis to the bottom, as you do when projecting points onto an image plane.
By setting location = [location[1],-location[2],location[0]], the rest of my code ran as expected.

@edgarriba

This comment has been minimized.

Copy link

edgarriba commented Nov 16, 2017

anyone faced: LogUnrealCV:Warning: error Argument Invalid
when calling vget /camera/0/proj_matrix ?

@jeffreykxiao

This comment has been minimized.

Copy link

jeffreykxiao commented Nov 21, 2017

@edgarriba as of the current version 0.3.10, the function vget /camera/0/proj_matrix is incomplete. It just contains a couple lines of commented out code and returns FExecStatus::InvalidArgument.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment