Road test: 60 km/h localization

[cyn-liu]

Hello!
We record some bags, which include the following topics:
/vehicle/status/velocity_status --wheel-speed
/imu/data --imu
/sensing/lidar/top/rectified/pointcloud --lidar-raw-scan
I plan use this bag to test the localization performance, so in your opinion, which dimension should I choose to evaluate Autoware's current localization pipe line? The calculation time, iteration times or ndt score?
Are all the evaluation dimension above enough?
@mitsudome-r @xmfcx do you have any suggestions?

[xmfcx]

@cyn-liu first of all, thanks for taking on this task! ٩(^ᴗ^)۶

My suggestion would be following:
Drive at:

• 20 km/h
• 40 km/h
• 60 km/h
  on the same route. I know it's impossible to drive on the exact same path but we should try.

This way we can compare how localization performance changes by velocity.

Also like you mentioned:

• calculation time per scan
• ndt score
  are ok.

It would be best to compare it to a gnss/ins rtk ground truth (if the environment is open enough for satellite visibility).

But since this data doesn't have it, maybe we can compare the "60 km/h" data by

• running localization in real time
• giving it extra time for each scan to achieve better accuracy (or running some more costly matching algorithms)
  to compare how velocity affects the performance.

[cyn-liu]

Thanks your replay! @xmfcx

[liuXinGangChina]

Hi, Fatih. @xmfcx
I think your suggestion is pretty resonable. But I think this test is mainly foucs on whether current localization stack is able to handle high speed AD scenarios . If the localization stack can do this, we plan to proceed more high way AD driving test in the future to push Autoware's edge to new horizon.

1. Currently the bag we collected contains several driving scenarios from 0km/h to 60km/h, and of course includes 20km/h and 40km/h. But if you intend to test with constant 20km/h or 40km/h, we haven't got that bag yet.
2. The test road is along a mountain, so the gnss signal is not signal even though we got RTK service by that region. So, we can't use rtk as Ground truth.
3. For the evaluation dimension, I'm totally agree with you
4. Regarding "running localization in real time under 60 km/h", i think it's not allowed by local regulation.
5. As for "giving it extra time for each scan to achieve better accuracy (or running some more costly matching algorithms)" i think we can try using other pointlcoud registration method provided by Autoware, and compare their performance.

[xmfcx]

@liuXinGangChina

Currently the bag we collected contains several driving scenarios from 0km/h to 60km/h, and of course includes 20km/h and 40km/h. But if you intend to test with constant 20km/h or 40km/h, we haven't got that bag yet.

It's ok to evaluate with what you've collected for now.

The test road is along a mountain, so the gnss signal is not signal even though we got RTK service by that region. So, we can't use rtk as Ground truth.

Ok, we need to rely on other point cloud matching algorithms for evaluation then.

Regarding "running localization in real time under 60 km/h", i think it's not allowed by local regulation.

I didn't mean to test it on the vehicle. I've meant, let's play the bag file in real time for analysis of runtime performance.

As for "giving it extra time for each scan to achieve better accuracy (or running some more costly matching algorithms)" i think we can try using other pointcloud registration method provided by Autoware, and compare their performance.

Let's try by giving point clouds more time to align first, if it's not good, we can try other methods like G-ICP or something similar later on. (To make first iteration of this entire evaluation process faster.)

[cyn-liu]

We recorded two bags with different speeds on the same test road and used these bags to evaluate Autoware's localization performance.

Point cloud map of test road:

Vehicle speed information in bag1:

Vehicle speed information in bag2:

Localization performance:

1. NDT matching score(nearest_voxel_transformation_likelihood):
   

2. NDT matching iteration number:
   

3. NDT matching execute time:
   

For detailed test results, please refer to this report:
Test report - localization performance.pdf

[n-patiphon]

@cyn-liu Thank you for sharing the evaluation results!

From the velocity plots, velocity seems to spike before the ends of both bags.
Were they just noise from odometry or the vehicle really did change its speed abruptly?

[cyn-liu]

This is the speed evaluated by EKF, so there is a lot of noise.