evaluation problem

[fishmarch]

Hi,
I followed the instructions in Installation and Eval. But when I run with
./tools/dist_test_map.sh ./projects/configs/maptr/maptr_tiny_r50_24e.py ./ckpts/maptr_tiny_r50_24e.pth 1

I get the following errors. Do you have any suggestions? Thanks.

projects.mmdet3d_plugin
/home/MapTR/projects/mmdet3d_plugin/bevformer/modules/custom_base_transformer_layer.py:94: UserWarning: The arguments `feedforward_channels` in BaseTransformerLayer has been deprecated, now you should set `feedforward_channels` and other FFN related arguments to a dict named `ffn_cfgs`. 
  warnings.warn(
/home/MapTR/projects/mmdet3d_plugin/bevformer/modules/custom_base_transformer_layer.py:94: UserWarning: The arguments `ffn_dropout` in BaseTransformerLayer has been deprecated, now you should set `ffn_drop` and other FFN related arguments to a dict named `ffn_cfgs`. 
  warnings.warn(
/home/MapTR/projects/mmdet3d_plugin/bevformer/modules/custom_base_transformer_layer.py:94: UserWarning: The arguments `ffn_num_fcs` in BaseTransformerLayer has been deprecated, now you should set `num_fcs` and other FFN related arguments to a dict named `ffn_cfgs`. 
  warnings.warn(
/root/miniconda3/envs/maptr/lib/python3.8/site-packages/mmcv/cnn/bricks/transformer.py:341: UserWarning: The arguments `feedforward_channels` in BaseTransformerLayer has been deprecated, now you should set `feedforward_channels` and other FFN related arguments to a dict named `ffn_cfgs`. 
  warnings.warn(
/root/miniconda3/envs/maptr/lib/python3.8/site-packages/mmcv/cnn/bricks/transformer.py:341: UserWarning: The arguments `ffn_dropout` in BaseTransformerLayer has been deprecated, now you should set `ffn_drop` and other FFN related arguments to a dict named `ffn_cfgs`. 
  warnings.warn(
/root/miniconda3/envs/maptr/lib/python3.8/site-packages/mmcv/cnn/bricks/transformer.py:341: UserWarning: The arguments `ffn_num_fcs` in BaseTransformerLayer has been deprecated, now you should set `num_fcs` and other FFN related arguments to a dict named `ffn_cfgs`. 
  warnings.warn(
/root/miniconda3/envs/maptr/lib/python3.8/site-packages/mmcv/cnn/bricks/transformer.py:92: UserWarning: The arguments `dropout` in MultiheadAttention has been deprecated, now you can separately set `attn_drop`(float), proj_drop(float), and `dropout_layer`(dict) 
  warnings.warn('The arguments `dropout` in MultiheadAttention '
load checkpoint from local path: ./ckpts/maptr_tiny_r50_24e.pth
The model and loaded state dict do not match exactly

unexpected key in source state_dict: pts_bbox_head.transformer.encoder.layers.0.attentions.1.attention.grid_offsets

[                                                  ] 0/6019, elapsed: 0s, ETA:Traceback (most recent call last):
  File "./tools/test.py", line 262, in <module>
    main()
  File "./tools/test.py", line 233, in main
    outputs = custom_multi_gpu_test(model, data_loader, args.tmpdir,
  File "/home/MapTR/projects/mmdet3d_plugin/bevformer/apis/test.py", line 70, in custom_multi_gpu_test
    for i, data in enumerate(data_loader):
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/MapTR/projects/mmdet3d_plugin/datasets/nuscenes_map_dataset.py", line 1238, in __getitem__
    return self.prepare_test_data(idx)
  File "/home/MapTR/projects/mmdet3d_plugin/datasets/nuscenes_map_dataset.py", line 1225, in prepare_test_data
    input_dict = self.get_data_info(index)
  File "/home/MapTR/projects/mmdet3d_plugin/datasets/nuscenes_map_dataset.py", line 1129, in get_data_info
    map_location = info['map_location'],
KeyError: 'map_location'

/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torch.distributed.run.
Note that --use_env is set by default in torch.distributed.run.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 12733) of binary: /root/miniconda3/envs/maptr/bin/python3
Traceback (most recent call last):
  File "/root/miniconda3/envs/maptr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/maptr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/run.py", line 689, in run
    elastic_launch(
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/maptr/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
***************************************
         ./tools/test.py FAILED        
=======================================
Root Cause:
[0]:
  time: 2023-03-22_13:47:54
  rank: 0 (local_rank: 0)
  exitcode: 1 (pid: 12733)
  error_file: <N/A>
  msg: "Process failed with exitcode 1"
=======================================
Other Failures:
  <NO_OTHER_FAILURES>
***************************************

[LegendBC]

Hi xiaoyu @fishmarch , thanks for your interest! The issue may lie in dataset preparation. You should generate the annotation with the script provided in this repo following the doc. Since we add the map location into the annotation dict here.

[fishmarch]

Thanks. I will have a check. I thought the data was well prepared by my colleagues, but unfortunately it was not...