Skip to content
semantic parser trained by using videos only instead of labeled logical forms
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
dist
experiments
lib/spf
scripts-for-setup
st-for-parser
.gitignore
README.md
compile.sh
format_experiment_results.py
pipeline.sh
reduce_sets.py
run.sh

README.md

Grounded semantic parser

This is the code for our paper at EMNLP 2018, Grounding language acquisition by training semantic parsers using captioned videos. Please cite us if you use our code or data and feel free to reach out with any questions.

All code and data are included in this repo except the videos; download here.

Running the Project

Data

There are two main experiment directories:
    visionparser/ (full supervision)
    groundedvisionparser (weak supervision that uses the videos and sentence tracker)

The training and test CCG files are identical except the weakly supervised format does not include the ground truth form.

To download videos, go to Dropbox.

Setting up sentence tracker

This is only necessary when training the model under weak supervision. If you intend to do a fully supervised run as a benchmark, or to use a saved model for either full or weak supervision, you can skip this step.

This tracker is from Seeing what you’re told: Sentence-guided activity recognition in video and A compositional framework for grounding language inference, generation, and acquisition in video.

Running Model

The script pipeline.sh builds the necessary Java code and runs a model from start to finish (this includes training and test). pipeline.sh has many required and optional arguments. Run pipeline.sh -h for a detailed description of arguments.

We write (most of) the experiment files from scratch. The EXPNAME.exp file is written from scratch. We write a unique file for each fold of the cross-validation, defining the training and test sets.

The libraries and model are recompiled.

There are a lot of options for pipeline.sh and they're described in the getopts section of the script.

sample commands:
./pipeline.sh -d visionparser -r RUNNAME -v s -t f (full supervision)
./pipeline.sh -d groundedvision parser -r RUNNAME -v s -o PORT -O 1 -t w (weak supervision)

Overview of directories

  • experiments
  • st-for-parser: sentence tracker

Editing Project

Definitely the easiest way to edit and run all of the Java code. This project requires Java 8+ (installation help).

Once Eclipse is installed, open and load the project. Add classpath variable

  • Window -> Preferences -> Java -> Build Path -> Classpath Variables
  • new variable named TINY_REPO; set path to lib/spf

Switch to workspace that differs from our project folder (e.g. eclipse-workspace).
Import project

  • select root directory (e.g. working-VisionParser)
  • import dependencies (select lib/spf & select nested directories)

Remove ignore line to run a test case; run as J1test.

We modify the Cornell SPF codebase- (a special thank you for publicly sharing your code!)

You can’t perform that action at this time.