INFER: INtermediate representations for FuturE pRediction
This repository contains code and data required to reproduce the results of INFER: INtermediate representations for FuturE pRediction (arXiv)
In order to use this code, you need to download the intermediate representations datasets used by our network. The intermediate representation has 5 different channels as shown in the above figure which are generated using the semantic and instance segmentation of the image along with depth. Most of the channels are self explanatory and the other vehicles channel represents the position of the other vehicles in the scene.
The intermediate representations have been generated for all the 3 datasets: KITTI, Cityscapes & Oxford RobotCar
The link to the dataset consisting of intermediate representations is given here
You can find the corresponding semantic, instance segmenation and disparity here
The code has been tested with
In order to install all the required files, create a virtualenv and install the files given in
virtualenv -p python3.5 venv source venv/bin/activate pip install -r requirements.txt
Running the demo scripts
In order to run our evaluation script for KITTI, run the file
infer-main.ipynb after changing the appropriate paths of
data_dir in the code.
repo_dir refers to the absolute path of the repository root in your PC.
data_dir refers to the absolute path of the corresponding dataset.
The scipts for transfer to Cityscapes & Oxford can be run in the same way. You need to run different scripts in order to evaluate the models on different datasets as shown below:
infer-main.ipynb: KITTI results
infer-transfer.ipynb: Cityscapes transfer
oxford-test.ipynb: Oxford RobotCar transfer
baseline.ipynb: Baseline KITTI results
baseline-transfer.ipynb: Baseline Cityscapes transfer results
Training the network on a single split of KITTI takes about 8-10 hours in an NVIDIA GeForce GTX 1080Ti GPU.
You can run the training code as follows:
python train.py -expID split-0 -nepochs 60 -dataDir /home/username/kitti -optMethod adam -initType default -lr 0.000100 -momentum 0.900000 -beta1 0.90000 -modelType skipLSTM -groundTruth True -imageWidth 256 -imageHeight 256 -scaleFactor False -gradClip 10 -seqLen 1 -csvDir /home/pravin.mali/merged/final-validation/ -trainPath train0.csv -valPath test0.csv -minMaxNorm False
The different parameters available are given in the file
args.py. In order to try the various ablation studies, you can set the arguments like
lane to be false
All other pretrained models are available on request.
For more plots, tables and project video refer to the project page here