No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

CLPS 1520 Project - Scene Labeling with Recurrent Convolutional Neural Networks


This project implements a "Recurrent Convolutional Neural Network" (rCNN) for scene labeling, as seen in Pinheiro et al, 2014.

Summary of files           -- README file
accuracy_results/   -- Text files containing results of different models
category_maps/      -- Text files containing category info for Stanford  & "Data from Games" datasets             -- Script for testing the model            -- Code for rCNN model
models/             -- Saved models from training    -- Code for processing input data and image patches
requirements.txt    -- Lists python package requirements
test_results/       -- Model output, as images            -- Script for trainin model


  1. Clone or download this repository to your computer:

    git clone

  2. Install the necessary Python requirements through pip:

    pip install -r requirements.txt

    This project only requires Tensorflow version 11, and PIL for image manipulation.

  3. Download a dataset with which to use the model. We used the Stanford Background Dataset which has 700+ 320x240 images and 8 classes. The code also works with the Data from Games dataset, which has 25,000 1914 × 1052 with 38 categories.

    To use another dataset, make sure it is organized similarly to one of the above two, and specify while training and testing which dataset it is "mimicking". Specifically, both datasets had the data in one folder, with subfolders "labels" and "images" for labels and images, respectively. The stanford dataset had labels in the format of space-separated digits in a text file, while the "Data from Games" dataset had labels in the form of paletted images, where each color corresponds to a different label.

  4. Generate a text file that maps colors to category numbers nad labels. Each line of the file has five space-separated values:

    R G B category_num category_id

Category files for the Stanford Background and Data From Games datasets are provided in the folder category_maps.



For training, use the script. The following command trains the model on the Stanford dataset:

python --dataset stanford-bground --category_map category_maps/stanford_bground_categories.txt --data_dir <STANFORD DATA FOLDER> --model_save_path <SAVED_MODEL_FILE>

Running -h will show additional parameters for the script, including different hyperparameters. also supports the ability to train on specific patches surrounding individual pixels, and to apply gaussian filters to the input patches.


For testing, use the script. This script will get per-class accuracies for each image, as well as output predicted labels as image files. The following command loads a saved model and evaluates accuracy on the stanford data set:

python --model <SAVED_MODEL> --category_map category_maps/stanford_bground_categories.txt --dataset stanford-bground --data_dir <STANFORD DATA FOLDER> --output_dir <RESULT IMAGES FOLDER>

Similar to, use the -h flag to see more parameters.


  • Tyler Barnes-Diana
  • Zhenhao Hou (Andrew)
  • Jae Hwan Hwang (James)
  • Raphael Kargon