Skip to content
[ACL 2019] Visually Grounded Neural Syntax Acquisition
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
demos
env [master] fix requirements Jul 8, 2019
src
.gitignore [master] init .gitignore Jun 25, 2019
LICENSE Initial commit May 22, 2019
README.md [master] update overview image Jul 10, 2019

README.md

Visually Grounded Neural Syntax Acquisition

Freda Shi, Jiayuan Mao, Kevin Gimpel, Karen Livescu

ACL 2019   [paper] [project page] [bib]

model.jpg

Requirements

PyTorch >= 1.0.1

See also env/conda_env.txt for detailed (but maybe not necessary) environment setup.

Data Preparation

Download our pre-processed data here and unzip it to the data folder.

To run our demo, the data folder should be organized as follows

data
├── mscoco
│   ├── dev_caps.txt
│   ├── dev_ims.npy
│   ├── test_caps.txt
│   ├── test_ground-truth.txt
│   ├── train_caps.txt
│   ├── train_ims.npy
│   └── vocab.pkl
...

Training

VG-NSL

bash demos/demo_train.sh

VG-NSL+HI

bash demos/demo_train_head-initial.sh

Inference/Testing

After training a model, we can test it by running

bash demos/demo_test.sh

Citation

If you found the codebase useful, please consider citing

@inproceedings{shi2019visually,
    Title = {Visually Grounded Neural Syntax Acquisition},
    Author = {Shi, Haoyue and Mao, Jiayuan and Gimpel, Kevin and Livescu, Karen},
    Booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
    Year = {2019}
}

Acknowledgement

The visual-semantic embedding part is adapted from the codebase of VSE++ (Faghri et al., BMVC 2018) and VSE-C (Shi et al., COLING 2018). Part of the basic code is adapated from Jiayuan's personal Python toolkits Jacinle and Freda's toolkits Witter. We also thank Victor Silva for providing the original concreteness estimation codebase (Hessel et al., NAACL-HLT 2018).

License

MIT

You can’t perform that action at this time.