Skip to content
Code for Episodic Memory Reader (EMR) https://arxiv.org/abs/1903.06164
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
babi
trivia
tvqa
README.md

README.md

Episodic Memory Reader (EMR)

This work has been published on The Meeting of the Association for Computational Linguistics (ACL 2019).
Here is the paper of arxiv version in https://arxiv.org/abs/1903.06164.
or you can find the paper of ACL anthology version in https://aclweb.org/anthology/P19-1434

Requirements

  • Python 3.6.4
  • Pytorch >= 1.0.0
  • tensorboardX >= 1.6
  • tqdm
  • termcolor
  • boto3
  • numpy
  • (below is for only tvqa)
  • pysrt
  • h5py
  • image
  • seaborn
  • matplotlib

How to use

Recommend running with at least three GPUs.

bAbI

cd babi  

To pre-process and train the model,

python main-a3c.py \
  --task-id=[2, 22] \
  --prepro \
  --model=[FIFO, UNIFORM, LIFO, LRU_DNTM, R_EMR, T_EMR] \
  --log-dir=$DIR  

You can see other configuration in 'main-a3c.py'.
If you have a pre-process, you do not have to do it again.

TriviaQA

cd trivia  

Download TriviaQA dataset from http://nlp.cs.washington.edu/triviaqa

To pre-process TriviaQA dataset,

python prepro.py  

To train the model,

python main-a3c.py \
  --task=[web, wikipedia] \
  --model=[FIFO, UNIFORM, LIFO, LRU_DNTM, R_EMR, T_EMR] \
  --rl-method=[a3c, policy, discrete] \
  --log-dir=$DIR  

You can see other configuration in 'main-a3c.py'.

Demo

python main-a3c.py --demo --ckpt=TRAINED_MODEL  

Test

python main-a3c.py --test --ckpt=TRAINED_MODEL  

TVQA

This code is mostly based on the original TVQA code (https://github.com/jayleicn/TVQA).

In case of Transformer code, it is based on the BERT code (https://github.com/huggingface/pytorch-pretrained-BERT).

cd tvqa

Dataset

To prepare the TVA dataset, please follow the instructions "Usage 1~3" of the original TVQA code (https://github.com/jayleicn/TVQA).

Codes for preprocessing are also included in this repository.

Preparing ImageNet feature is also needed.

In case of ImageNet feature, please divide it in to pieces using "sep_large.py".

Usage:

  1. Pretraining
    python main-a3c.py --pretrain \
    	--log-dir=$PRE_DIR
    

Pretraining the TVQA model is needed to get right results.

  1. Training

    python main-a3c.py --input_streams sub imagenet \
    	--no_ts \
    	--pretrain-dir=$PRE_DIR \
    	--model=[FIFO, UNIFORM, LIFO, LRU_DNTM, R_EMR, T_EMR] \
    	--log-dir=$DIR
    
  2. Test

    python main-a3c.py --test --ckpt=$DIR
    
  3. Demo

    python main-a3c.py --demo --ckpt=$DIR
    

Acknowledgements

This work was supported by Clova, NAVER Corp. and Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (No.2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion).

You can’t perform that action at this time.