No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data_prepare remove random crop in data augmentations Jan 18, 2018
data_providers add sideway path for segmentation, as well as segmentation visualization Apr 27, 2018
models add sideway path for segmentation, as well as segmentation visualization Apr 27, 2018
.gitignore add versioning for action mapping arch Nov 12, 2017
README.md Update README.md Nov 29, 2017
__init__.py change some training params, use custom to implement datadriven conti… Aug 24, 2017
batching.py better loss functions, help debugging Feb 20, 2018
config.py change some training params, use custom to implement datadriven conti… Aug 24, 2017
dataset.py
eval.py add sideway path for segmentation, as well as segmentation visualization Apr 27, 2018
gather_stat.py some final polishments Aug 17, 2017
setup.sh update Readme to enable better reproducibility and add the model zoo Aug 29, 2017
train.py small fixes Nov 2, 2017
util.py modify Jul 15, 2017
util_car.py add sideway path for segmentation, as well as segmentation visualization Apr 27, 2018
wrapper.py enable extra post config functions Jan 24, 2018
wrapper_test.ipynb polish the visualization to be good to look Sep 4, 2017

README.md

Berkeley DeepDrive (BDD) Driving Model

Project Introduction:

Within BDD Driving Project, we formulate the self driving task as future egomotion prediction. To attack the task, we collected Berkeley DeepDrive Video Dataset with our partner Nexar, proposed a FCN+LSTM model and implement it using tensorflow.

The Berkeley DeepDrive Video Dataset(BDD-V)

BDD-V dataset will be released here.

Using Our Code:

Installation

First clone the codebase to your local file system at $BDD_ROOT.

git clone https://github.com/gy20073/BDD_Driving_Model.git && cd BDD_Driving_Model && export BDD_ROOT=$(pwd)

For Ubuntu 14.04 and 16.04, you can install all dependencies using:

cd $BDD_ROOT && bash setup.sh

Or if you don't want to install Anaconda or you're using other versions of Linux, you could manually install those packages:

  • Tensorflow 0.11
  • ffmpeg and ffprobe
  • Python packages:
    • IPython, PIL, opencv-python, scipy, matplotlib, numpy, sklearn

Model Zoo

We provide some pretrained models that are ready to use. Download the model zoo here to $BDD_ROOT"/data" and make sure they are available at locations like $BDD_ROOT"/data/discrete_cnn_lstm/model.ckpt-146001.bestmodel".

The tf.caffenet.bin is a pretrained Alexnet model generated from the Caffe-Tensorflow tool. It's used as the finetuning start point for the driving model.

Run a Pretrained Model

With the pretrained model, you could test it on your own dashcam video. wrapper.py is a simple wrapper that use the model without the requirement to prepare a TFRecord dataset. It takes in an image at every 1/3 second and output the predicted future egomotion. See wrapper_test.ipynb for an example usage.

TODO(Improve Visualization)

Data Preparation:

Download and unzip the dataset's training and validation set into some directory $DATA_ROOT. There should be directories like $DATA_ROOT/train/videos and $DATA_ROOT/val/info.

Then run the following commands to generate indexes of videos and convert raw videos to TFRecords. For validation set:

cd $BDD_ROOT"/data_prepare"
python filter.py $DATA_ROOT/val
python prepare_tfrecords.py --video_index=$DATA_ROOT/val/video_filtered_38_60.txt --output_directory=$DATA_ROOT/tfrecords/validation

and on the training set:

cd $BDD_ROOT"/data_prepare"
python filter.py $DATA_ROOT/train
python prepare_tfrecords.py --video_index=$DATA_ROOT/train/video_filtered_38_60.txt --output_directory=$DATA_ROOT/tfrecords/train

Model Training, Validation and Monitoring:

To train a driving model, first change some path flags in $BDD_ROOT"/config.py". In particular set FLAGS.pretrained_model_path = "$BDD_ROOT/data/tf.caffenet.bin" and FLAGS.data_dir = "$DATA_ROOT/tfrecords". They are paths to the ImageNet pretrained Alexnet model and the TFRecord files we got from the previous data preparation step.

There are a bunch of different types of models proposed in the paper and implemented in this repo. The configuration of each model is a function in config.py, such as discrete_tcnn1 and continuous_datadriven_bin. The discrete_tcnn1 model is a model with temporal convolution of window size 1 and the model predicts discrete driving actions such as Go, Stop, Left and Right. The continuous_datadriven_bin model is a CNN-LSTM style model that predicts continuous egomotions, including future angular velocity and future speed. The binning method used in this model is a data-driven approach.

We will use discrete_tcnn1 as a running example, the training procedures of other models are similar. To train the model, run

cd $BDD_ROOT && python config.py train discrete_tcnn1

For the continuous_datadriven_bin model, we need to get the distribution before do the actually training, to get the distribution, run

cd $BDD_ROOT && python config.py stat continuous_datadriven_bin

During training, the program will write checkpoints and logs to $BDD_ROOT"/data/discrete_tcnn1". To monitor the validation performance, we could start another process evaluating the model periodically

cd $BDD_ROOT && python config.py eval discrete_tcnn1

One could also use tensorboard to visually monitor the training progress

cd $BDD_ROOT"/data" && tensorboard --logdir=. --port=8888

and open it at: http://localhost:8888

Reference:

If you want to cite our paper, please use the following bibtex:

@article{xu2016end,
  title={End-to-end learning of driving models from large-scale video datasets},
  author={Xu, Huazhe and Gao, Yang and Yu, Fisher and Darrell, Trevor},
  journal={arXiv preprint arXiv:1612.01079},
  year={2016}
}