Representation learning for VisualSem knowledge graph.
- Requirements
- Python 3.6+
- PyTorch 1.1.0+
- dgl-cu100 0.4.3
- Data
- Graph:
- Features:
- GraphSage+DistMult:
- Training:
- To train our best model using image+text features and node+edge gating:
python train.py - To train model without image or text features:
python train.py --mode node - To train model using only image features and node+edge gating:
python train.py --mode img - To train model using only text features and node+edge gating:
python train.py --mode gl - To modify the number of negative samples (default: 1000):
python train.py --neg_sample_size 100 --batch_size 10000
- To train our best model using image+text features and node+edge gating:
- Evaluation:
- Download pretrain model weights here and unzip it. Place it at the same folder as
eval.py - To evaluate our best model using image+text features and node+edge gating:
python eval.py - To evaluate model without image or text features:
python eval.py --mode node - To evaluate model using only image features and node+edge gating:
python eval.py --mode img - To evaluate model using only text features and node+edge gating:
python eval.py --mode gl - To modify the number of negative samples (default: 1000):
python eval.py --neg_sample_size 100 --batch_size 10000
- Download pretrain model weights here and unzip it. Place it at the same folder as
Please access this Colab Notebook and follow the instructions in it to run a BERT-based token classification (NER) model on data from the WNUT-17 task.
You will also need to download this .zip file and upload it to Colab. The notebook contains instructions about this file.
The example in the notebook above walks through the WNUT-17 task, but you may change the {train/dev/test}.txt data files to a task of your choice. Note that this also requires you to retrieve nodes from the VisualSem KG that correspond to your task data. Code for node retrieval can be found in the VisualSem git repository (see: Retrieval). Retrieved nodes for the WNUT-17 task are provided in the .zip file above.
- Requirements
- Python 3.6+
- PyTorch 1.1.0+
- transformers-3.5.0
- Data
- To download raw images in MultiSense dataset please follow the link here. Alternatively, run commands:
pip install gdowngdown https://drive.google.com/uc?id=1e0ebK7KWlBzlc0j2u3CpXWJ0zVupPxM9 -O YOURLOCATION!tar xvf multiSenseImagesAll.tar.gz
- To download other files used in training, please follow the link [here](TBA - Iacer)
- Reference file for verb, query phrase and its German translation:
gold_german_query_classes.csv - ResNet152 image features for train/valid/test sets:
features_per_image_train_german.h5/features_per_image_val_german.h5/features_per_image_test_german.h5 - Look-up table for (image_path, image_name, image_verb) for train/valid/test sets:
train_german.pkl/val_german.pkl/test_german.pkl - Look-up table for verb to integer index (based on training set):
verb_map.pkl - Look-up table for top-1 retrieved node hidden state for each query:
query_nodes.pkl - Node hidden state files: (TBA - from Yash's gdrive)
- Reference file for verb, query phrase and its German translation:
- Training/Evaluatoin
- To train our baseline:
python multi_train.py --epochs 10 --num_layer 2 --projection --lr 5e-4 --dropout 0.1 --nonlinear --hidden_dim 128 - To train with node hidden states:
python multi_train.py --epochs 10 --node --num_layer 2 --projection --lr 5e-4 --dropout 0.1 --nonlinear --hidden_dim 100 - To specify different node hidden state files, add argument:
--node_file "/content/drive/My Drive/graph/nyu_multimodal_kb/NER/graph_emb_img.t"
- Misc
- Script for creating ResNet152 image features:
img_feature.py - Model checkpoints: (TBA - gdrive)