Refine ZED-Velodyne calibration #179

aaguiar96 · 2020-06-13T11:32:30Z

Now that we have the objective function working, we should:

Recalibrate the ZED camera intrinsics
Record a new dataset

eupedrosa · 2020-06-13T11:36:58Z

Following #149, @aaguiar96 that rotation the ZED camera have may be cause by that -83 in the P matrix. Try to redo the calibration now, but using the K matrix to see if anything changes.

aaguiar96 · 2020-06-13T11:40:37Z

Following #149, @aaguiar96 that rotation the ZED camera have may be cause by that -83 in the P matrix. Try to redo the calibration now, but using the K matrix to see if anything changes.

Ok, but since, the -83 is not correct right?
I'll go to the lab on Monday and try to recalibrate the camera and record a new dataset.

eupedrosa · 2020-06-13T11:43:40Z

You do not need a new dataset, you can use the one that you have right now. The K matrix does not have that -83.

miguelriemoliveira · 2020-06-13T11:50:57Z

Hi @aaguiar96, when you say "Record a new dataset" you mean record a new bag file, right?

@eupedrosa , if @aaguiar96 is going to the lab he could indeed record a new bag file because the one we have right now has several problems (#157 )

eupedrosa · 2020-06-13T11:52:07Z

I known!! I was just curious to see if we already have a working solution...

aaguiar96 · 2020-06-13T11:56:33Z

I known!! I was just curious to see if we already have a working solution...

I don't think it's working...

aaguiar96 · 2020-06-13T11:57:25Z

Hi @aaguiar96, when you say "Record a new dataset" you mean record a new bag file, right?

@eupedrosa , if @aaguiar96 is going to the lab he could indeed record a new bag file because the one we have right now has several problems (#157 )

Yes, I don't know if I'll have time to solve all the issues since I have to perform the calibration, but at least I can record a longer dataset, with the pattern closer to the camera.

eupedrosa · 2020-06-13T11:57:49Z

Whitout the camera I cannot judge :\ It is not visible. Can you add the tf tree in the visualization?

aaguiar96 · 2020-06-13T12:29:28Z

Whitout the camera I cannot judge :\ It is not visible. Can you add the tf tree in the visualization?

The camera is there.

eupedrosa · 2020-06-13T12:33:10Z

Just to be clear, this calibration uses the K matrix or the `P' matrix?

aaguiar96 · 2020-06-13T12:33:48Z

Just to be clear, this calibration uses the K matrix or the `P' matrix?

The K matrix...

aaguiar96 · 2020-06-15T11:06:57Z

Calibration result:

Left:

image_width: 1280
image_height: 720
camera_name: narrow_stereo/left
camera_matrix:
  rows: 3
  cols: 3
  data: [652.25783,   0.     , 670.14407,
           0.     , 652.16532, 365.16427,
           0.     ,   0.     ,   1.     ]
camera_model: plumb_bob
distortion_coefficients:
  rows: 1
  cols: 5
  data: [-0.196334, 0.034772, -0.002054, 0.003261, 0.000000]
rectification_matrix:
  rows: 3
  cols: 3
  data: [ 0.99760051,  0.00946769, -0.06858274,
         -0.00884404,  0.99991679,  0.00939135,
          0.06866594, -0.00876227,  0.99760123]
projection_matrix:
  rows: 3
  cols: 4
  data: [672.44807,   0.     , 822.27319,   0.     ,
           0.     , 672.44807, 366.49005,   0.     ,
           0.     ,   0.     ,   1.     ,   0.     ]

Right:

image_width: 1280
image_height: 720
camera_name: narrow_stereo/right
camera_matrix:
  rows: 3
  cols: 3
  data: [672.6178 ,   0.     , 650.11811,
           0.     , 672.20899, 380.37149,
           0.     ,   0.     ,   1.     ]
camera_model: plumb_bob
distortion_coefficients:
  rows: 1
  cols: 5
  data: [-0.233312, 0.078484, -0.003954, 0.002391, 0.000000]
rectification_matrix:
  rows: 3
  cols: 3
  data: [ 0.99575785,  0.00898342, -0.09157295,
         -0.00981562,  0.99991449, -0.00864153,
          0.09148749,  0.00950371,  0.99576087]
projection_matrix:
  rows: 3
  cols: 4
  data: [672.44807,   0.     , 822.27319, -78.89036,
           0.     , 672.44807, 366.49005,   0.     ,
           0.     ,   0.     ,   1.     ,   0.     ]

aaguiar96 · 2020-06-15T11:16:09Z

Btw, the -78 appears again. This might me the translation of the right camera in relation with the left one...

I doubt that the manufactury calibration is wrong...

eupedrosa · 2020-06-15T11:25:58Z

I also doubt it is wrong. But we should only care about the K matrix. However, for that to work, we need to have access to the unrectified images.

The image the ZED sdk returns, is it rectified or not? or it publishes both?

miguelriemoliveira · 2020-06-15T11:49:45Z

Hi, That was an advance. Great work! How about a comparison between the K matrix now and the k matrix we had before? Also, to know if the calibration was any good can you tell us the reprojection error (printed by the ros calibration) as well as the number of images? Recording the non rectified is important...

…

On Mon, Jun 15, 2020, 12:26 Eurico F. Pedrosa ***@***.***> wrote: I also doubt it is wrong. But we should only care about the K matrix. However, for that to work, we need to have access to the unrectified images. The image the ZED sdk returns, is it rectified or not? or it publishes both? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#179 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTHVSK7SG5RJ6WRM5UKVTRWYANJANCNFSM4N46MZ2Q> .

aaguiar96 · 2020-06-15T12:49:39Z

I also doubt it is wrong. But we should only care about the K matrix. However, for that to work, we need to have access to the unrectified images.

The image the ZED sdk returns, is it rectified or not? or it publishes both?

It publishes both. Tomorrow I will record both. I guess that the one we're using is rectified...
Can that be a problem on the optimization?

aaguiar96 · 2020-06-15T12:51:28Z

Also, to know if the calibration was any good can you tell us the reprojection error (printed by the ros calibration) as well as the number of images?

The calibration have 52 images for each camera. I did not save the reprojection error, I don't know it now...

miguelriemoliveira · 2020-06-15T13:09:21Z

It publishes both. Tomorrow I will record both. I guess that the one we're using is rectified...
Can that be a problem on the optimization?

Hi @aaguiar96 , you should record only the raw (unrectified image). It is a problem for the optimization because we would be undistorting and undistorted image.

aaguiar96 · 2020-06-15T13:10:24Z

Hi @aaguiar96 , you should record only the raw (unrectified image). It is a problem for the optimization because we would be undistorting and undistorted image.

Ok so... This could improve the result! Tomorrow I'll fix this! :)

aaguiar96 · 2020-06-19T10:03:10Z

I @eupedrosa

How did you overcome the blocking problem when saving a collection?
It's happening to me, even with bag_rate = 0.5 ...

Here's my command:

roslaunch agrob_calibration collect_data.launch output_folder:=$ATOM_DATASETS bag_rate:=0.5 overwrite:=true

aaguiar96 · 2020-06-19T10:03:39Z

Btw, the rosbag looks really nice this time! :)

aaguiar96 · 2020-06-19T10:15:26Z

Another thing, did you adjust the initial_estimate?

The robot is really misaligned with the pattern points. See here (for a frame where I am in front of the camera):

eupedrosa · 2020-06-19T10:17:10Z

How did you overcome the blocking problem when saving a collection?

It has something to do with difference of timestamps. I'm just trying to fix it and I'll will push it.

Another thing, did you adjust the initial_estimate?

Yes I did.

aaguiar96 · 2020-06-19T10:18:26Z

It has something to do with difference of timestamps. I'm just trying to fix it and I'll will push it.

Ok... But do you think that they are too different? They just differ on the nsecs parameter...

eupedrosa · 2020-06-19T10:31:05Z

@aaguiar96, I solved it. You cal pull the change. It was a typo in the key used in the config dictionary.

This is something fragil in our code. There is no hardening against this kinda stuff.

aaguiar96 · 2020-06-29T16:08:49Z

New dataset taken from the most recent bag file.

29-06.zip

aaguiar96 · 2020-07-01T09:47:48Z

Hi @miguelriemoliveira and @eupedrosa

Here's the (link) new set of bagfiles that consider the new approach of having the pattern at ~45º.
I'll try out the optimization with them in the afternoon.

I have started yesterday thinking about that problem on the LiDAR beam clustering.
I tried to convert the 2D projection that we have of each LiDAR points in the pattern to an image to apply the Hough transform, but I'm afraid that this approach will fail sometimes...

I'll try out something more brute force, a clustering based on a distance threshold for example.
When I have some news I'll let you know.

miguelriemoliveira · 2020-07-01T10:00:53Z

Hi André, Thanks. I will download. About the labelling I also have some ideas. If you want we can discuss in the afternoon? 14h30? 15h? Miguel * Miguel Riem de Oliveira * email: mriem@ua.pt | m.riem.oliveira@gmail.com office: 22.2.18 | ext: 23887 Dep. de Engenharia Mecânica, Universidade de Aveiro *Campus Universitário de Santiago, 3810-193 Portugal*

…

On Wed, 1 Jul 2020 at 10:48, André Aguiar ***@***.***> wrote: Hi @miguelriemoliveira <https://github.com/miguelriemoliveira> and @eupedrosa <https://github.com/eupedrosa> Here's the (link <https://drive.google.com/file/d/1vJuBApSFSqjgWRJK47A7wwvQBK_ZucPd/view?usp=sharing>) new set of bagfiles that consider the new approach of having the pattern at ~45º. I'll try out the optimization with them in the afternoon. I have started yesterday thinking about that problem on the LiDAR beam clustering. I tried to convert the 2D projection that we have of each LiDAR points in the pattern to an image to apply the *Hough transform*, but I'm afraid that this approach will fail sometimes... I'll try out something more brute force, a clustering based on a distance threshold for example. When I have some news I'll let you know. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#179 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWTHVWAYJX6SXJA7UJSK2LRZMA5FANCNFSM4N46MZ2Q> .

aaguiar96 · 2020-07-01T10:50:32Z

About the labelling I also have some ideas. If you want we can discuss in the afternoon? 14h30? 15h?

14h30 is fine by me @miguelriemoliveira, deal! :)

aaguiar96 · 2020-07-01T13:38:51Z

Hi @miguelriemoliveira and @eupedrosa

I recorded a dataset with one of the today's bagfiles.
Here it is.
dataset-01-17-2020.zip

I'll try it out now,

aaguiar96 · 2020-07-01T15:22:25Z

Here:

roslaunch agrob_calibration calibrate.launch dataset_file:=/home/andre-criis/Documents/saved_datasets/29-06/data_collected.json csf:="lambda name: int(name) < 3

…ag run_calibration on the calibrate.launch which allows to run the system without running the optimization, which can then be launched in a separate window (better for debugging)

miguelriemoliveira · 2020-07-01T16:23:09Z

Hi @aaguiar96 ,

Check the latest commit. It automatically creates the labeled_data rviz marker.

aaguiar96 · 2020-07-01T17:54:01Z

Hi @aaguiar96 ,

Check the latest commit. It automatically creates the labeled_data rviz marker.

Hi @miguelriemoliveira, great, I'll pull now.

Good news, the labeling procedure is working with the solution you proposed.
Here are two images, one with the dataset that I was showing yesterday in the meeting, and other with the dataset that I recorded today. In the last one, it is quite visible the correct labeling.

Thanks for the help @miguelriemoliveira :-)

aaguiar96 · 2020-07-01T17:58:17Z

@eupedrosa just for you to know, @miguelriemoliveira proposed a labeling solution considering the spherical representation [r, theta, phi] of the velodyne data.
The beams are clustered using the theta component, and the limits are the maximum and minimum value of the phi component for each cluster.

…ation of point cloud

aaguiar96 · 2020-07-01T18:04:05Z

@miguelriemoliveira, I committed the changes so that we're synchronized.
Now we just have to solve the mystery of the optimization!

eupedrosa · 2020-07-01T20:57:50Z

Great! The rotated pattern seems to provide more information.

… up the objective_function.py before. Now its fixed and easier.

…rmalization

eupedrosa · 2020-07-02T16:47:51Z

Any news? Should we keep this issue open?

…eems to be working fine.

miguelriemoliveira · 2020-07-02T23:00:31Z

Hi @aaguiar96 and @eupedrosa

I was working with @eupedrosa after working with @aaguiar96 and we looked into the normalization. It is not yet complete but I think the results have improved very much. The previous normalization had a bug in some collections which could be catastrophic.

Also, we added the optimization of the intrinsics and the distortion components (distortion was not there) since there images are clearly in need of some distortion. After this, camera errors droped from 2.5 pixels avg to bellow 1 pixel.

Then we ran a full agrob system calibration . Here is the result with some 8 collections

https://youtu.be/6yydzfB6UBM

Then I ran with all 20 collections

https://youtu.be/k5B0wRdWcpo

Conclusions:

All laser data points are near the patterns, where they should, both orthogonally and longitudinally.

Point cloud avg error 0.008 meters, bellow 1 centimeter
Image avg error 0.99 pixels, bellow 1 pixel.
Velodyne is tilted upward as it should (to compensate for the tild downward that the zed has but is not in the xacro)
Velodyne z seems to be fine
Velodyne is shifted to the right ... that's the only thing that still bothers me.

So to me this looks really good, the only thing is 5. @aaguiar96 can you take (or ask someone to take some pictures, and even measurements with a metric tape).

If the velodyne is really in the center of the camera we can resort to the best calibration technique: grab a hammer and hit the sensor from left to right until it goes to the position estimated in hte optimization :)

aaguiar96 · 2020-07-02T23:24:32Z

Uau, looks really really great.

So, the problem was on the nornalization and intrinsics right? How do you added the distortion components since they were not available on the camera info?

I forgot to share, but I took some pictures of the sensors. Here:

So, now we just have to check and try to measure the distance between the calibration and the real sensor configuration.
I'll ask someone tomorrow to measure the distances with a hammer. :)

Other thing, did you optimize the three sensors or just a single camera and the lidar?

eupedrosa · 2020-07-03T09:32:28Z

From I'm seeing the calibration is working !! I'm more inclined to say the URDF is wrong, not the calibration.
@aaguiar96, I think you should focus on getting the correct pose of the velodyne on the robot and use it as the anchored sensor.

Furthermore, I think we can finally close this long issue.

aaguiar96 · 2020-07-03T09:41:27Z

👏 good work team! :-)

Ok, I think you're right.
Closing this one, and opening another to focus on that.

miguelriemoliveira · 2020-07-03T11:50:08Z

Hi guys. I agree, we make a very good team! And we are almost there.

But I am not sure I am so optimistic as @eupedrosa ... perhaps, but I mean, from the picture we have this yellow distance

and from the optimization we have

The question is: is this pose of the velodyne w.r.t. the camera consistent with reality? That will not change if we fix the velodyne instead of the camera I think ...

miguelriemoliveira · 2020-07-03T11:53:55Z

One thing that does not make sense is the size of the zed camera ... the cad model seems too small. @aaguiar96 can you check it?

eupedrosa · 2020-07-03T11:59:35Z

I am a little more optimistic because our eyes easily lie to us. Obviously, we should check.

aaguiar96 · 2020-07-03T12:00:35Z

The best thing to do is measure it on rviz and physically...
I can do that on monday! And also I'll take a look at the URDFs.

miguelriemoliveira · 2020-07-03T12:00:37Z

Ok, I hope you are right :) another thing is that there is a second camera there? It is not being used right?

aaguiar96 · 2020-07-03T12:18:15Z

another thing is that there is a second camera there? It is not being used right?

No. The other camera is a stereo realsense. It has two fisheye len cameras.

aaguiar96 added the enhancement New feature or request label Jun 13, 2020

aaguiar96 assigned miguelriemoliveira and aaguiar96 Jun 13, 2020

aaguiar96 mentioned this issue Jun 18, 2020

Ideas for next agrob bagfile #157

Closed

aaguiar96 added a commit that referenced this issue Jul 1, 2020

#179 - New approach for labelling procedure using spherical represent…

6700ceb

…ation of point cloud

miguelriemoliveira added a commit that referenced this issue Jul 2, 2020

#179 found the problem with the lidar optimization. Probably I messed…

3b38564

… up the objective_function.py before. Now its fixed and easier.

aaguiar96 added a commit that referenced this issue Jul 2, 2020

#179 - beam distance to orthogonal distance and velodyne residuals no…

0562978

…rmalization

miguelriemoliveira added a commit that referenced this issue Jul 2, 2020

#179. Improved a lot by looking at the normalization. Now it almost s…

16cff01

…eems to be working fine.

aaguiar96 closed this as completed Jul 3, 2020

aaguiar96 mentioned this issue Jul 3, 2020

Calibrated zed camera does not have distortion #197

Closed

Refine ZED-Velodyne calibration #179

Refine ZED-Velodyne calibration #179

Comments

aaguiar96 commented Jun 13, 2020

eupedrosa commented Jun 13, 2020

aaguiar96 commented Jun 13, 2020

eupedrosa commented Jun 13, 2020

miguelriemoliveira commented Jun 13, 2020

eupedrosa commented Jun 13, 2020

aaguiar96 commented Jun 13, 2020

aaguiar96 commented Jun 13, 2020

eupedrosa commented Jun 13, 2020 • edited Loading

aaguiar96 commented Jun 13, 2020

eupedrosa commented Jun 13, 2020

aaguiar96 commented Jun 13, 2020

aaguiar96 commented Jun 15, 2020

aaguiar96 commented Jun 15, 2020

eupedrosa commented Jun 15, 2020

miguelriemoliveira commented Jun 15, 2020 via email

aaguiar96 commented Jun 15, 2020

aaguiar96 commented Jun 15, 2020

miguelriemoliveira commented Jun 15, 2020

aaguiar96 commented Jun 15, 2020

aaguiar96 commented Jun 19, 2020

aaguiar96 commented Jun 19, 2020 • edited Loading

aaguiar96 commented Jun 19, 2020 • edited Loading

eupedrosa commented Jun 19, 2020

aaguiar96 commented Jun 19, 2020

eupedrosa commented Jun 19, 2020

aaguiar96 commented Jun 29, 2020

aaguiar96 commented Jul 1, 2020

miguelriemoliveira commented Jul 1, 2020 via email

aaguiar96 commented Jul 1, 2020

aaguiar96 commented Jul 1, 2020

aaguiar96 commented Jul 1, 2020

miguelriemoliveira commented Jul 1, 2020

aaguiar96 commented Jul 1, 2020

aaguiar96 commented Jul 1, 2020

aaguiar96 commented Jul 1, 2020

eupedrosa commented Jul 1, 2020

eupedrosa commented Jul 2, 2020

miguelriemoliveira commented Jul 2, 2020

aaguiar96 commented Jul 2, 2020 • edited Loading

eupedrosa commented Jul 3, 2020

aaguiar96 commented Jul 3, 2020

miguelriemoliveira commented Jul 3, 2020

miguelriemoliveira commented Jul 3, 2020

eupedrosa commented Jul 3, 2020

aaguiar96 commented Jul 3, 2020

miguelriemoliveira commented Jul 3, 2020

aaguiar96 commented Jul 3, 2020

eupedrosa commented Jun 13, 2020 •

edited

Loading

aaguiar96 commented Jun 19, 2020 •

edited

Loading

aaguiar96 commented Jun 19, 2020 •

edited

Loading

aaguiar96 commented Jul 2, 2020 •

edited

Loading