Grounded semantic parser
This is the code for our paper at EMNLP 2018, Grounding language acquisition by training semantic parsers using captioned videos. Please cite us if you use our code or data and feel free to reach out with any questions.
All code and data are included in this repo except the videos; download here.
Running the Project
There are two main experiment directories:
visionparser/ (full supervision)
groundedvisionparser (weak supervision that uses the videos and sentence tracker)
The training and test CCG files are identical except the weakly supervised format does not include the ground truth form.
To download videos, go to Dropbox.
Setting up sentence tracker
This is only necessary when training the model under weak supervision. If you intend to do a fully supervised run as a benchmark, or to use a saved model for either full or weak supervision, you can skip this step.
pipeline.sh builds the necessary Java code and runs a model from start to finish (this includes training and test).
pipeline.sh has many required and optional arguments. Run
pipeline.sh -h for a detailed description of arguments.
We write (most of) the experiment files from scratch. The EXPNAME.exp file is written from scratch. We write a unique file for each fold of the cross-validation, defining the training and test sets.
The libraries and model are recompiled.
There are a lot of options for
pipeline.sh and they're described in the getopts section of the script.
./pipeline.sh -d visionparser -r RUNNAME -v s -t f (full supervision)
./pipeline.sh -d groundedvision parser -r RUNNAME -v s -o PORT -O 1 -t w (weak supervision)
Overview of directories
st-for-parser: sentence tracker
Definitely the easiest way to edit and run all of the Java code. This project requires Java 8+ (installation help).
Once Eclipse is installed, open and load the project. Add classpath variable
Window -> Preferences -> Java -> Build Path -> Classpath Variables
- new variable named TINY_REPO; set path to lib/spf
Switch to workspace that differs from our project folder (e.g. eclipse-workspace).
- select root directory (e.g. working-VisionParser)
- import dependencies (select lib/spf & select nested directories)
Remove ignore line to run a test case; run as J1test.
We modify the Cornell SPF codebase- (a special thank you for publicly sharing your code!)