Skip to content

Commit

Permalink
Initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@KaimingHe
KaimingHe committed Apr 9, 2018
0 parents commit 4b5d2dd
Show file tree
Hide file tree
Showing 76 changed files with 517,816 additions and 0 deletions.
3 changes: 3 additions & 0 deletions CODE_OF_CONDUCT.md
@@ -0,0 +1,3 @@
# Code of Conduct

Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the [full text](https://code.facebook.com/pages/876921332402685/open-source-code-of-conduct) so that you can understand what actions will and will not be tolerated.
39 changes: 39 additions & 0 deletions CONTRIBUTING.md
@@ -0,0 +1,39 @@
# Contributing to video-nonlocal-net
We want to make contributing to this project as easy and transparent as
possible.

## Our Development Process
... (in particular how this is synced with internal changes to the project)

## Pull Requests
We actively welcome your pull requests.

1. Fork the repo and create your branch from `master`.
2. If you've added code that should be tested, add tests.
3. If you've changed APIs, update the documentation.
4. Ensure the test suite passes.
5. Make sure your code lints.
6. If you haven't already, complete the Contributor License Agreement ("CLA").

## Contributor License Agreement ("CLA")
In order to accept your pull request, we need you to submit a CLA. You only need
to do this once to work on any of Facebook's open source projects.

Complete your CLA here: <https://code.facebook.com/cla>

## Issues
We use GitHub issues to track public bugs. Please ensure your description is
clear and has sufficient instructions to be able to reproduce the issue.

Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
disclosure of security bugs. In those cases, please go through the process
outlined on that page and do not file a public issue.

## Coding Style
* 2 spaces for indentation rather than tabs
* 80 character line length
* ...

## License
By contributing to video-nonlocal-net, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.
74 changes: 74 additions & 0 deletions DATASET.md
@@ -0,0 +1,74 @@
# Dataset Preparation

## Download our copy

We have prepared one copy of the Kinetics dataset. To obtain the dataset, please email xiaolonw@cs.cmu.edu. Our copy includes 234643 training and 19761 validation videos of the Kinetics dataset. All the videos are rescaled to height=256 pixels via ffmpeg, thus the total size is much smaller: 132GB.

Because of the expirations of some YouTube links, we can only download 234,643 out of 246,535 videos for training (on 12/20/2017), so about 10K (**5%**) training videos are missing. If you are training with this data, it will lead to a slight performance drop (<0.5%) compared to the numbers reported in our paper.

**Note:** In this repo, we release the models that are trained with the same data as our paper. However, this data with fewer missing videos is currently unavailable.

## Download via the official code

One can also download the videos via the official code. This is how it is done:

1. Download the videos via the official [scripts](https://github.com/activitynet/ActivityNet/tree/master/Crawler/Kinetics).

2. After all the videos are downloaded on the path: YOUR_DATASET_FOLDER. Go into the folder:
```Shell
cd process_data/kinetics
```

We provide a dictionary for the corresponding name of each class
```Shell
classids.json
```

Use the following script to generate the txt lists for both training and validation set. This script will also modify the names of some folders in the dataset (e.g., "petting cat" -> "petting_cat").
```Shell
gen_py_list.py
```

Use the following script to rescale the videos to height=256 pixels. This is not necessary, but it will make the IO much faster during training.
```Shell
downscale_video_joblib.py
```

## Creating lmdb for training and testing

In this code, we are going to perform video decoding and extract the video frames on the fly during training. Following the instruction below, we can prepare the lmdb for training.
```Shell
cd process_data/kinetics
```

1. This step is only necessary if you are using our copy of the Kinetics data. Otherwise, "trainlist.txt" and "vallist.txt" should have already been generated by following [`Download via the official code`](DATASET.md#download-via-the-official-code). If you downloaded our copy, we need to change the folder name in the train/val lists, assuming the dataset is downloaded in YOUR_DATASET_FOLDER, run:
```Shell
python change_listname.py trainlist_download.txt trainlist.txt /nfs.yoda/xiaolonw/kinetics/data/train $YOUR_DATASET_FOLDER/train_256
python change_listname.py vallist_download.txt vallist.txt /nfs.yoda/xiaolonw/kinetics/data/val $YOUR_DATASET_FOLDER/val_256
```

2. Since lmdb does not support shuffle during training, We shuffle the training list and repeat the shuffling for 100 times (as 100 epochs). The results are saved in one single txt file (2GB). It is crucial to shuffle after each epoch when training with Batch Normalization.
```Shell
python shuffle_list_rep.py trainlist_shuffle_rep.txt
```

3. Create the lmdb according to the list. Note that the lmdb is only storing the file names instead of the videos themselves. The test set is the same as the validation set. During training, the validation error is measured on random cropped examples. During testing, we perform FCN (spatial) test without flipping by default.
```Shell
mkdir ../../data/lmdb
mkdir ../../data/lmdb/kinetics_lmdb_multicrop
python create_video_lmdb.py --dataset_dir ../../data/lmdb/kinetics_lmdb_multicrop/train --list_file trainlist_shuffle_rep.txt
python create_video_lmdb.py --dataset_dir ../../data/lmdb/kinetics_lmdb_multicrop/val --list_file vallist.txt
python create_video_lmdb_test_multicrop.py --dataset_dir ../../data/lmdb/kinetics_lmdb_multicrop/test --list_file vallist.txt
```

4. (Optional:) This is the lmdb for FCN (spatial) testing with flipping.
```Shell
mkdir ../../data/lmdb/kinetics_lmdb_flipcrop
python create_video_lmdb_test_flipcrop.py --dataset_dir ../../data/lmdb/kinetics_lmdb_flipcrop/test --list_file vallist.txt
```

5. (Optional:) This is the lmdb for testing by only using center cropped examples.
```Shell
mkdir ../../data/lmdb/kinetics_lmdb_singlecrop
python create_video_lmdb_test.py --dataset_dir ../../data/lmdb/kinetics_lmdb_singlecrop/test --list_file vallist.txt
```
110 changes: 110 additions & 0 deletions INSTALL.md
@@ -0,0 +1,110 @@
# Installing Caffe2

## Getting Caffe2 from Xiaolong's Fork

The experiments in this code are done with Xiaolong's personal fork of caffe2, one can obtain it by
```Shell
git clone --recursive https://github.com/xiaolonw/caffe2
```

## Getting Official Caffe2

To obtain the newest ops in caffe2 (e.g., [Detectron](https://github.com/facebookresearch/Detectron)), one can download the official caffe2 and then replace the video folder in caffe2 with our customized_ops.
```Shell
git clone --recursive https://github.com/caffe2/caffe2
rm -rf caffe2/caffe2/video
cp -r caffe2-video-nlnet/caffe2_customized_ops/video caffe2/caffe2/
```

## Installation following Official Instructions

Before installing caffe2, one should also install ffmpeg with anaconda.
```Shell
conda install -c conda-forge ffmpeg=3.2.4=3
```
Also modify and set the option "USE_FFMPEG" in caffe2/CMakeLists.txt "ON"
```Shell
option(USE_FFMPEG "Use ffmpeg" ON)
```

To fully install Caffe2 with CUDA support, follow the [installation instructions](https://caffe2.ai/docs/getting-started.html) from the [Caffe2 website](https://caffe2.ai/).

After installation, remember to adjust your `PYTHONPATH` environment variable to include its location (`/path/to/caffe2/build/install`, where `build` is the Caffe2 CMake build directory) as well as the location of our code (`/path/to/caffe2-video-nlnet/lib`)
```Shell
export PYTHONPATH=/path/to/caffe2/build/install:/path/to/caffe2-video-nlnet/lib:$PYTHONPATH
```

## Installation Example

If you are still confused about the installation, we now give an installation example which Xiaolong used in school (cluster with Red Hat Linux OS). The installation can be done via [Anaconda]('https://www.anaconda.com/download/#linux') without sudo access.

Starting from installed Anaconda, we suggest you to create a virtual environment first, so that the new packages will not conflict with your existing libs:
```Shell
conda create -n caffe2 python=2.7
conda activate caffe2
```

Then we can install all the required packages via "conda install":
```Shell
conda install --yes cmake && \
conda install --yes git && \
conda install --yes glog && \
conda install --yes gflags && \
conda install --yes gcc

conda install cudnn=6.0.21=cuda8.0_0
conda install -c conda-forge ffmpeg=3.2.4=3

conda install --yes opencv && \
conda install --yes networkx && \
conda install --yes cython && \
conda install --yes libpng && \
conda install --yes protobuf && \
conda install --yes flask && \
conda install --yes future && \
conda install --yes graphviz && \
conda install --yes hypothesis && \
conda install --yes pydot && \
conda install --yes lmdb && \
conda install --yes pyyaml && \
conda install --yes matplotlib && \
conda install --yes requests && \
conda install --yes scipy && \
conda install --yes setuptools && \
conda install --yes six && \
conda install --yes tornado
```

Modify your bashrc by adding (assuming "/home/USERNAME/anaconda2/" is directory for anaconda):
```Shell
export PATH=/home/USERNAME/anaconda2/envs/caffe2/bin:/usr/local/bin:/usr/local/cuda-8.0/bin:$PATH
export C_INLCUDE_PATH=/home/USERNAME/anaconda2/envs/caffe2/include:/usr/local/cuda-8.0/include:$C_INLCUDE_PATH
export CPLUS_INLCUDE_PATH=/home/USERNAME/anaconda2/envs/caffe2/include:/usr/local/cuda-8.0/include:$CPLUS_INLCUDE_PATH
export LD_LIBRARY_PATH=/home/USERNAME/anaconda2/envs/caffe2/lib:/usr/lib64:/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=/home/USERNAME/anaconda2/envs/caffe2/lib:/usr/lib64:/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/lib:$LIBRARY_PATH
conda activate caffe2
```

Download Caffe2:
```Shell
git clone --recursive https://github.com/xiaolonw/caffe2
```

Remember to modify and set the option "USE_FFMPEG" in caffe2/CMakeLists.txt "ON"
```Shell
option(USE_FFMPEG "Use ffmpeg" ON)
```

Install Caffe2 (assuming "/path/to/caffe2/" is directory for Caffe2):
```Shell
cd /path/to/caffe2
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX:PATH=/path/to/caffe2/build/install ..
make -j16 install
```

After installation, remember to adjust your `PYTHONPATH` environment variable to include its location (`/path/to/caffe2/build/install`) as well as the location of our code (`/path/to/caffe2-video-nlnet/lib`) in bashrc:
```Shell
export PYTHONPATH=/path/to/caffe2/build/install:/path/to/caffe2-video-nlnet/lib:$PYTHONPATH
```

0 comments on commit 4b5d2dd

Please sign in to comment.