Skip to content
Branch: master
Go to file
Code

Latest commit

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Grounded semantic parser

This is the code for our paper at EMNLP 2018, Grounding language acquisition by training semantic parsers using captioned videos. Please cite us if you use our code or data and feel free to reach out with any questions.

All code and data are included in this repo except the videos; download here.

Running the Project

Data

There are two main experiment directories:
    visionparser/ (full supervision)
    groundedvisionparser (weak supervision that uses the videos and sentence tracker)

The training and test CCG files are identical except the weakly supervised format does not include the ground truth form.

To download videos, go to Dropbox.

Setting up sentence tracker

This is only necessary when training the model under weak supervision. If you intend to do a fully supervised run as a benchmark, or to use a saved model for either full or weak supervision, you can skip this step.

This tracker is from Seeing what you’re told: Sentence-guided activity recognition in video and A compositional framework for grounding language inference, generation, and acquisition in video.

Running Model

The script pipeline.sh builds the necessary Java code and runs a model from start to finish (this includes training and test). pipeline.sh has many required and optional arguments. Run pipeline.sh -h for a detailed description of arguments.

We write (most of) the experiment files from scratch. The EXPNAME.exp file is written from scratch. We write a unique file for each fold of the cross-validation, defining the training and test sets.

The libraries and model are recompiled.

There are a lot of options for pipeline.sh and they're described in the getopts section of the script.

sample commands:
./pipeline.sh -d visionparser -r RUNNAME -v s -t f (full supervision)
./pipeline.sh -d groundedvision parser -r RUNNAME -v s -o PORT -O 1 -t w (weak supervision)

Overview of directories

  • experiments
  • st-for-parser: sentence tracker

Editing Project

Definitely the easiest way to edit and run all of the Java code. This project requires Java 8+ (installation help).

Once Eclipse is installed, open and load the project. Add classpath variable

  • Window -> Preferences -> Java -> Build Path -> Classpath Variables
  • new variable named TINY_REPO; set path to lib/spf

Switch to workspace that differs from our project folder (e.g. eclipse-workspace).
Import project

  • select root directory (e.g. working-VisionParser)
  • import dependencies (select lib/spf & select nested directories)

Remove ignore line to run a test case; run as J1test.

We modify the Cornell SPF codebase- (a special thank you for publicly sharing your code!)

About

semantic parser trained by using videos only instead of labeled logical forms

Topics

Resources

Releases

No releases published
You can’t perform that action at this time.