Skip to content
Permalink
Browse files

Release of the datasets and models

  • Loading branch information...
rodrigoberriel committed Dec 22, 2017
1 parent e071078 commit 72a22e97c996b0df126917e62be28411af1e79dc
@@ -1,5 +1,73 @@
## Automatic Large-Scale Data Acquisition via Crowdsourcing for Crosswalk Classification: A Deep Learning Approach
# Automatic Large-Scale Data Acquisition via Crowdsourcing for Crosswalk Classification: A Deep Learning Approach

Accepted for publication in [Computers & Graphics](https://www.journals.elsevier.com/computers-and-graphics/).
[Rodrigo F. Berriel](http://rodrigoberriel.com), Franco Schmidt Rossi, [Alberto F. de Souza](https://inf.ufes.br/~alberto), and [Thiago Oliveira-Santos](https://www.inf.ufes.br/~todsantos/home)

More information soon. In the meantime, check [this page](http://www.lcad.inf.ufes.br/wiki/index.php/Automatic_Large-Scale_Data_Acquisition_via_Crowdsourcing_for_Crosswalk_Classification:_A_Deep_Learning_Approach).
[Computers & Graphics](https://www.journals.elsevier.com/computers-and-graphics): [10.1016/J.CAG.2017.08.004](http://dx.doi.org/10.1016/J.CAG.2017.08.004)

[![Overview](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/overview.png)](http://www.sciencedirect.com/science/article/pii/S0097849317301334)

Correctly identifying crosswalks is an essential task for the driving activity and mobility autonomy. Many crosswalk classification, detection and localization systems have been proposed in the literature over the years. These systems use different perspectives to tackle the crosswalk classification problem: satellite imagery, cockpit view (from the top of a car or behind the windshield), and pedestrian perspective. Most of the works in the literature are designed and evaluated using small and local datasets, i.e. datasets that present low diversity. Scaling to large datasets imposes a challenge for the annotation procedure. Moreover, there is still need for cross-database experiments in the literature because it is usually hard to collect the data in the same place and conditions of the final application. In this paper, we present a crosswalk classification system based on deep learning. For that, crowdsourcing platforms, such as OpenStreetMap and Google Street View, are exploited to enable automatic training via automatic acquisition and annotation of a large-scale database. Additionally, this work proposes a comparison study of models trained using fully-automatic data acquisition and annotation against models that were partially annotated. Cross-database experiments were also included in the experimentation to show that the proposed methods enable use with real world applications. Our results show that the model trained on the fully-automatic database achieved high overall accuracy (94.12%), and that a statistically significant improvement (to 96.30%) can be achieved by manually annotating a specific part of the database. Finally, the results of the cross-database experiments show that both models are robust to the many variations of image and scenarios, presenting a consistent behavior.

---

### Test with your own data
Pre-trained models are available [here](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/models/).

This [Python notebook](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/inference/) may help you with the inference process.


### Datasets
We are releasing three datasets: IARA (day and night) and GOPRO. The IARA datasets were recorded using the infrastructure of the IARA autonomous car (which we are developing in our lab), and the GOPRO dataset was recorded using a GOPRO camera attached to the windshield (close to the rear mirror). Altogether, these three datasets have more than 35,000 annotated images. More details can be found in the paper. Below, you can see some samples of the datasets and how to download each of them.

### IARA dataset


##### Daylight

Details and download [here](https://github.com/rodrigoberriel/streetview-crosswalk-classification/tree/master/datasets#iara-daylight).

![SamplesIARAdaylight](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/samples-iara.png)

Results on the IARA dataset: [video](https://youtu.be/zrKU3duNwuo).

[![ResultsIARA](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/youtube-iara.png)](https://youtu.be/zrKU3duNwuo)


##### Night

Details and download [here](https://github.com/rodrigoberriel/streetview-crosswalk-classification/tree/master/datasets#iara-night).

![SamplesIARAnight](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/samples-iara-night.png)

Results on the NIGHT dataset: [video](https://youtu.be/afCBi1Pj1NE).

[![ResultsNIGHT](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/youtube-iara-night.png)](https://youtu.be/afCBi1Pj1NE)


### GOPRO dataset

Details and download [here](https://github.com/rodrigoberriel/streetview-crosswalk-classification/tree/master/datasets#gopro).

![SamplesGOPRO](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/samples-gopro.png)

Results on the GOPRO dataset: [video](https://youtu.be/jmYmQFiqY3c).

[![ResultsGOPRO](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/youtube-gopro.png)](https://youtu.be/jmYmQFiqY3c)


#### BibTeX

If you find any of the datasets and/or our pre-trained models useful for your research, please cite the paper below.

@article{berriel2017cag,
Author = {Rodrigo F. Berriel and Franco Schmidt Rossi and Alberto F. de Souza and Thiago Oliveira-Santos},
Title = {{Automatic Large-Scale Data Acquisition via Crowdsourcing for Crosswalk Classification: A Deep Learning Approach}},
Journal = {Computers \& Graphics},
Year = {2017},
Volume = {68},
Pages = {32--42},
DOI = {10.1016/J.CAG.2017.08.004},
ISSN = {0097-8493},
}

If any link is broken, feel free to send me an email or open an issue.
@@ -0,0 +1,33 @@
## Datasets

Three datasets are being released:

- IARA (daylight)
- IARA (night)
- GOPRO

Altogether, they have more than 35,000 images in ~40GB.

#### IARA (daylight)

The IARA dataset is named after the vehicle used to capture the images: the Intelligent Autonomous Robotic Automobile. IARA is an autonomous vehicle that is being developed in the High Performance Computing Lab (LCAD) of the Universidade Federal do Espírito Santo. The vehicle contains many sensors, but only one camera was used to record this dataset. The camera, a Bumblebee XB3, was mounted on the top of the car facing forward. This dataset was recorded during the day in a week-day, i.e. it contains usual traffic. In total, there are 12,441 images from 4 different sequences recorded in the capital of the Espírito Santo, Vitória.

![SamplesIARAdaylight](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/samples-iara.png)

Download [here](https://drive.google.com/drive/folders/11ETzxerQnfDdxkOSvX9LWKyyxRoqT7Wr?usp=sharing). The images are in the four `.tar.gz` (images1, images2, images3, and images4) and the annotations in the `iara.txt` file.

#### IARA (night)

The sequence comprises 12,114 frames (in a temporal sequence) covering part of the IARA dataset and more (including a toll, a bridge, etc.) during the night.

![SamplesIARAnight](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/samples-iara-night.png)

Download [here](https://drive.google.com/drive/folders/1cg0R5GpNBbtr2Sps8I6CUFrDRRseS-sr?usp=sharing). The images are in the `iara-night.tar.gz` and the annotations in the `iara-night.txt` file.

#### GOPRO

The GOPRO dataset comprises 11,070 images recorded in the city of Vitória, Vila Velha and Guarapari, Espírito Santo, Brazil. The videos were recorded using a GoPRO HERO 3 camera in Full HD (1920 × 1080 pixels) at 29.97 frames per second (FPS) in different days. Some of them were recorded in city roads and the others in the highways connecting these cities. The images are divided into 29 sequences. From them, 23 of them are short sequences (up to 15s) of a vehicle passing by crosswalks and the other 6 are longer sequences (up to 90 s) of a vehicle driving without any crosswalk in the field of view. The crosswalks in these sequences are presented in a variety of ways, such as with pedestrians, occluded by cars, painting fading away (i.e. aging), etc.

![SamplesGOPRO](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/images/samples-gopro.png)

Download [here](https://drive.google.com/drive/folders/1x2FWpCpzv8TLwAGigRa3eEbWcvr3E5xT?usp=sharing). The images are in the `WITH_CROSSWALK.tar.gz` (the sequences that have at least one crosswalk) and `WITHOUT_CROSSWALK.tar.gz` (the sequences without any crosswalk); and the annotations in the `gopro.txt` file.
BIN +179 KB images/overview.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

Large diffs are not rendered by default.

@@ -0,0 +1,15 @@
## Models

We're releasing two pre-trained models (the ones you could see in the videos):

- GSV-FA\*: the best out of the 10 models trained using fully-automatic dataset
- GSV-PA\*: the best out of the 10 models trained using partially-automatic dataset

You can find both models [here](https://drive.google.com/drive/folders/1dTFPQ8g2pA0M9G6zzOAYnboecqv4iUTE?usp=sharing). If the link is broken, feel free to send me an email or open an issue.

For each model, you'll find:
- Weights: `{model_name}.caffemodel`
- Mean: `mean.binaryproto` and `mean.jpg`
- Metadata: `{model_name}.metadata` (internal stuff, don't worry)

To test these models with your own data, you can follow the Python notebook provided [here](https://github.com/rodrigoberriel/streetview-crosswalk-classification/blob/master/inference/).

0 comments on commit 72a22e9

Please sign in to comment.
You can’t perform that action at this time.