Cognitive Transfuser

**Official Repository of Multimodal Data Fusion for Waypoint Prediction of Autonomous Vehicle **

Overview

This is an official implementation of cognitive transfuser. Cognitive transfuser is an improved version of transfuser; utilizing auxiliary networks for traffic light classification and real-time semantic segmentation. The figure and table below show the overall structure and experiment results of cognitive transfuser respectively.

---

Setup

Get requirements

git clone https://github.com/Hwansoo-Choi/Cognitive-Transfuser/edit/Cognitive-Transfuser
conda create -n cognitive_transfuser python=3.7
conda activate cognitive_transfuser
pip3 install -r requirements.txt

Get real-time semantic segmentation module

cd ..
git clone https://github.com/MichaelFan01/STDC-Seg

Download pretrained model from STDC-Seg.

Locate the STDC1-Seg and STDC2-Seg directories at

STDC-Seg/checkpoints/

Setup CARLA

chmod +x setup_carla.sh
./setup_carla.sh

Get dataset

Please refer to Dataset and Data Generation parts of Transfuser.

Get pretrained model

You can download pretrained model of cognitive transfuer from here.

Please locate the 'best_model.pth' file in Cognitive-Transfuser/cognitive_transfuser/log/ directory.

---

Train

cd cognitive_transfuser
python train.py

The trained model is saved in cognitive_transfuser/log/

---

Evaluate

Running CARLA server

Before running evaluation code, we must run CARLA server first.

SDL_VIDEODRIVER=offscreen SDL_HINT_CUDA_DEVICE=0 ./CarlaUE4.sh --world-port=2000 -opengl

Running evaluation code

sh /leaderboard/scripts/cognitive_transfuser_eval.sh

Acknowledgement

This implementation is based on CARLA Autopilot Leaderboard and Transfuser.

Semantic segmentation module and pretrained model are from STDC-Seg.