Skip to content
Scene Graph Prediction with Limited Labels
Jupyter Notebook Python Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Scene Graph Prediction with Limited Labels


  • Scene graphs capture visual relationships between objects in images, e.g. <person, riding, bike>. They have emerged as useful in a number of computer vision tasks, including visual question answering and captioning.

  • However, most scene graph datasets are sparse due to annotator error!

  • This work attempts to overcome limitations of human annotators using a semi-supervised method, taking advantage of both limited labels and unlabeled data, to generate training datasets for scene graphs.

  • Approach:

    • We leverage image-agnostic features, which are cheap to extract given bounding box pairs of images.
    • With as few as n=10 labeled relationships per predicate (e.g. "walk", "ride", "eat"), we learn heuristics, shallow decision trees that can serve as noisy labelers.
    • Because these heuristics are error prone, we learn a generative model to combine and denoise the outputs of these heuristics, producing probabilistic labels for each object pair.
    • Using probabilistic labels over the unlabeled training data, we can bootstrap training for any downstream scene graph model!


Please run the following script to download the VisualGenome dataset.


To create the virtual environment with appropriate requirements:

virtualenv -p python3 .env
source .env/bin/activate
pip install -r requirements.txt


All instructions for this demonstration are included in main.ipynb.


You can find the full paper corresponding to this work, presented at ICCV 2019, at


Please refer to the following citation if you are building on this work:

    author = {Chen, Vincent S. and Varma, Paroma and Krishna, Ranjay and Bernstein, Michael and Re, Christopher and Fei-Fei, Li},
    title = {Scene Graph Prediction With Limited Labels},
    booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
    month = {October},
    year = {2019}


Feel free to open an issue or send an email if you have any questions!

You can’t perform that action at this time.