checkpoint results

[zjuluolun]

Hello,
I'm using the official checkpoint to run test and find the scores of "AP_ped" is much lower than the one reported in the log. My reproduced result and the logged results are respectively,
{'mAP': 0.19033822417259216, 'AP_ls': 0.31963453, 'AP_ped': 0.061041933, 'TOP_lsls': 0.2537}
{'mAP': 0.3345, 'AP_ls': 0.31965008, 'AP_ped': 0.349356800, 'TOP_lsls': 0.2538}.
The detailed outputs of my test is

++ date +%y%m%d.%H%M%S
+ timestamp=240124.211800
+ WORK_DIR=work_dirs/lanesegnet
+ CONFIG=projects/configs/lanesegnet_r50_8x1_24e_olv2_subset_A.py
+ CHECKPOINT=work_dirs/lanesegnet/lanesegnet_r50_8x1_24e_olv2_subset_A.pth
+ GPUS=2
+ PORT=28510
+ python -m torch.distributed.run --nproc_per_node=2 --master_port=28510 tools/test.py projects/configs/lanesegnet_r50_8x1_24e_olv2_subset_A.py work_dirs/lanesegnet/lanesegnet_r50_8x1_24e_olv2_subset_A.pth --launcher pytorch --out-dir work_dirs/lanesegnet/test --eval openlane_v2
+ tee work_dirs/lanesegnet/test.240124.211800.log
WARNING:__main__:*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
load checkpoint from local path: work_dirs/lanesegnet/lanesegnet_r50_8x1_24e_olv2_subset_A.pth
load checkpoint from local path: work_dirs/lanesegnet/lanesegnet_r50_8x1_24e_olv2_subset_A.pth
[>>>>>>>>>>>>>>>>>>>>>>>>>>] 4806/4806, 15.3 task/s, elapsed: 313s, ETA:     0s2024-01-24 21:26:00,097 - mmdet - INFO - Starting format results...


2024-01-24 21:34:24,142 - mmdet - INFO - Starting openlanev2 evaluate...
calculating distances:: 100%|███████████████| 4806/4806 [19:57<00:00,  4.01it/s]
{'mAP': 0.19033822417259216, 'AP_ls': 0.31963453, 'AP_ped': 0.061041933, 'TOP_lsls': 0.2537472264730754}

Can you help me to find out whether I configure something wrong?

[sephyli]

Please check your data version, the data of LaneSegNet has a little difference with OpenLane-V2 Map Element Bucket. You may need to use tools/data_process.py to regenerate the correct data.

The ped crossing in LaneSegNet dataset follows the lane segment style, whereas in the OpenLane-V2 Map Element Bucket, it is represented using a polygon style.

[zjuluolun]

The problem has been solved after updating to the latest dataset. Many Thanks!

[Dchenhj]

hello， I'm facing the same issue.

I found that my annotation file contains the following content, but I am not ensure that my dataset is the most recent one ：
"meta_data": {
"source": "ArgoverseV2",
"source_id": "00a6ffc1-6ce9-3bc3-a060-6006e9893a1a"
}

[sephyli]

Hi, based on the information provided, it is difficult to identify the version of your dataset. If you downloaded the dataset before October 2023, there is a possibility that it is outdated.

To resolve this issue, we recommend redownloading the latest Map Element Bucket data. The download size is quite small, approximately 240MB.