Evaluation results on nuScenes using pretrained weight

[BoLang615]

Hi Junjie,

I appreciate your excellent work. I am trying to evaluate the provided BEVDet-Tiny model on nuScenes val set.
The command was "bash ./tools/dist_test.sh configs/bevdet/bevdet-sttiny.py checkpoints/bevdet-sttiny-pure.pth 1 --eval bbox --out ./workdirs/bevdey-sttiny-eval-results.pkl". And I got the following results:

mAP: 0.2751
mATE: 0.7179
mASE: 0.2738
mAOE: 0.5512
mAVE: 0.8749
mAAE: 0.2206
NDS: 0.3737
Eval time: 120.9s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.441 0.631 0.167 0.131 1.037 0.254
truck 0.197 0.757 0.225 0.125 0.828 0.227
bus 0.283 0.680 0.185 0.139 1.895 0.350
trailer 0.132 1.053 0.224 0.463 0.547 0.068
construction_vehicle 0.066 0.795 0.484 1.174 0.095 0.358
pedestrian 0.301 0.788 0.305 1.320 0.848 0.412
motorcycle 0.235 0.704 0.262 0.612 1.438 0.090
bicycle 0.182 0.607 0.265 0.875 0.310 0.006
traffic_cone 0.445 0.616 0.333 nan nan nan
barrier 0.468 0.547 0.287 0.122 nan nan

I am wondering why I could not get the reported mAP values (30.8). Did I miss something here?
Thank you.

[HuangJunJie2017]

@BoLang615 I think there is something about the python environment and here is the pip list I use. I suggest you check the version of the key modules like OpenCV, scikit-image, torch, and NumPy. Wish you can fix this problem and offer me some feedback so that I can prevent this potential problem.

absl-py==0.13.0
addict==2.4.0
anyio==3.6.1
argcomplete==1.12.3
argon2-cffi==21.1.0
astor==0.8.1
attrs==21.2.0
Babel==2.10.1
backcall==0.2.0
backports.entry-points-selectable==1.1.0
beautifulsoup4==4.11.1
black==21.10b0
bleach==4.1.0
brotlipy==0.7.0
cached-property==1.5.2
cachetools==4.2.2
ccimport==0.3.7
certifi==2021.5.30
cffi @ file:///tmp/build/80754af9/cffi_1625814693446/work
cfgv==3.3.1
chardet @ file:///tmp/build/80754af9/chardet_1607706768982/work
click==8.0.3
conda==4.10.3
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1618262151086/work
cryptography @ file:///tmp/build/80754af9/cryptography_1616769182610/work
cumm==0.2.8
cycler==0.10.0
Cython==0.29.24
debugpy==1.4.3
decorator==5.1.0
defusedxml==0.7.1
deprecation==2.1.0
descartes==1.1.0
distlib==0.3.3
entrypoints==0.3
fastjsonschema==2.15.3
filelock==3.3.2
fire==0.4.0
flake8==4.0.1
gast==0.2.2
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.40.0
h5py==3.6.0
identify==2.3.3
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
imageio==2.10.1
importlib-metadata==4.2.0
iniconfig==1.1.1
ipykernel==6.4.1
ipython==7.27.0
ipython-genutils==0.2.0
ipywidgets==7.6.5
jedi==0.18.0
Jinja2==3.1.2
joblib==1.0.1
json5==0.9.8
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==7.0.3
jupyter-console==6.4.0
jupyter-core==4.8.1
jupyter-packaging==0.12.1
jupyter-server==1.17.0
jupyterlab==3.4.2
jupyterlab-pygments==0.1.2
jupyterlab-server==2.14.0
jupyterlab-widgets==1.0.2
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.3.2
lark==1.1.2
llvmlite==0.31.0
lmdb==1.3.0
lyft-dataset-sdk==0.0.8
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib==3.4.3
matplotlib-inline==0.1.3
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.3.0
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626179032232/work
mkl-service==2.4.0
mmcv-full==1.3.13
mmdet==2.14.0
mmpycocotools==12.0.3
mmsegmentation==0.14.1
motmetrics==1.1.3
mypy-extensions==0.4.3
nbclassic==0.3.7
nbclient==0.5.4
nbconvert==6.5.0
nbformat==5.4.0
nest-asyncio==1.5.1
networkx==2.2
ninja==1.10.2.3
nodeenv==1.6.0
notebook==6.4.4
notebook-shim==0.1.0
numba==0.48.0
numpy==1.18.0
nuscenes-devkit==1.1.7
oauthlib==3.1.1
olefile==0.46
open3d==0.9.0.0
opencv-python==4.5.3.56
opt-einsum==3.3.0
packaging==21.0
pandas==1.3.4
pandocfilters==1.5.0
parso==0.8.2
pathspec==0.9.0
pccm==0.3.4
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.4.0
platformdirs==2.4.0
plotly==5.3.1
pluggy==1.0.0
plyfile==0.7.4
portalocker==2.4.0
pre-commit==2.15.0
prettytable==2.3.0
prometheus-client==0.11.0
prompt-toolkit==3.0.20
protobuf==3.18.0
ptyprocess==0.7.0
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.9.2
pycocotools==2.0.1
pycodestyle==2.8.0
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pyflakes==2.4.0
Pygments==2.10.0
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing==2.4.7
pyquaternion==0.9.9
pyrsistent==0.18.0
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
pytest==6.2.5
python-dateutil==2.8.2
pytz==2021.3
PyWavelets==1.1.1
PyYAML==5.4.1
pyzmq==22.3.0
qtconsole==5.1.1
QtPy==1.11.1
regex==2021.11.2
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
requests-oauthlib==1.3.0
rsa==4.7.2
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016701961/work
scikit-image==0.18.3
scikit-learn==0.24.2
scipy==1.4.1
Send2Trash==1.8.0
Shapely==1.7.1
six @ file:///tmp/build/80754af9/six_1623709665295/work
sniffio==1.2.0
soupsieve==2.3.2.post1
tenacity==8.0.1
tensorboard==2.1.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow-estimator==2.1.0
tensorflow-gpu==2.1.0
termcolor==1.1.0
terminado==0.12.1
terminaltables==3.1.10
testpath==0.5.0
threadpoolctl==2.2.0
tifffile==2021.11.2
timm==0.4.12
tinycss2==1.1.1
toml==0.10.2
tomli==1.2.2
tomlkit==0.11.0
torch==1.9.0
torchaudio==0.9.0a0+33b2469
torchvision==0.10.0
tornado==6.1
tqdm @ file:///tmp/build/80754af9/tqdm_1625563689033/work
traitlets==5.1.0
trimesh==2.35.39
typed-ast==1.4.3
typing-extensions @ file:///Users/ktietz/demo/mc3/conda-bld/typing_extensions_1629887998905/work
urllib3 @ file:///tmp/build/80754af9/urllib3_1625084269274/work
virtualenv==20.10.0
waymo-open-dataset-tf-2-1-0==1.2.0
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.2
Werkzeug==2.0.1
widgetsnbextension==3.5.1
wrapt==1.13.3
xmltodict==0.13.0
yacs==0.1.8
yapf==0.31.0
zipp==3.5.0

[BoLang615]

I git pull the updates from your latest commits and try to evaluate again.
Now, I could get the same results as the report.

mAP: 0.3082
mATE: 0.6647
mASE: 0.2729
mAOE: 0.5329
mAVE: 0.8286
mAAE: 0.2053
NDS: 0.4036
Eval time: 119.3s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.508 0.535 0.159 0.127 0.947 0.232
truck 0.222 0.671 0.216 0.123 0.834 0.220
bus 0.311 0.760 0.195 0.086 1.592 0.301
trailer 0.150 0.986 0.229 0.443 0.518 0.054
construction_vehicle 0.073 0.720 0.482 1.093 0.103 0.342
pedestrian 0.336 0.738 0.301 1.326 0.861 0.409
motorcycle 0.262 0.704 0.262 0.595 1.450 0.075
bicycle 0.213 0.525 0.270 0.885 0.325 0.009
traffic_cone 0.506 0.518 0.331 nan nan nan
barrier 0.502 0.490 0.284 0.119 nan nan

[BoLang615]

And I have another question about velocity regression. In BEVDet, the inputs of the network are the multi-view images in the current frame, and there is no previous frame information, how does the network accurately predict the velocity of the targets in the current labeled frame？

[HuangJunJie2017]

@BoLang615 You can refer to BVEDet4D for more accurate velocity prediction. As 0.8286 mAVE of BEVDet is a relatively poor performance in the literature when compared with BEVDet4D(~0.3 mAVE) or CenterPoint(~0.3 mAVE Lidar).

[BoLang615]

Thank you for the reply.