Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation
by Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Zhengdao Li, Long Hu, Kai Hwang.
This repository is for our Findings of NAACL 2022 paper 'Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation '. In this paper, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the instruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4.
- Dataset can be downloaded in Google Drive.
- Our trained model can be downloaded in Google Drive. If the trained model doesn't work or if there are any issues, please feel free to contact us.
- The pre-trained model can be downloaded in bert-base-german-dbmdz-uncased.
pip install --editable .
Note that, if the download speed is not fast, try this:
pip install --editable . -i https://pypi.tuna.tsinghua.edu.cn/simple
Our code is implemented over Python 3.6.8, and Pytorch 1.5.0+cu101. (We didn't test the code on other package version.)
pip install torch==1.5.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
If you use our prepared dataset, skip this step. Else:
cd preprocessing
python prepare_data.py
Then you can get the following files in your destination dir:
Choose one training method (with / without automl).
cd trainer
# please config train.sh first, and then:
sh train.sh
Config automl/config.yml and automl/search_space.json files, and run the following cmd to train in your terminal:
nnictl create --config automl/config.yml -p 11111
# -p means the port you wish to visualize the training process in browser.
If succeed, you should see the following logs in your terminal:
Go to your browser to see the training process.
Please refer to NNI Website for more instructions on NNI.
cd postprocessing
sh get_bleu4.sh
Please contact yongcao_epic@hust.edu.cn.
1.'ascii' codec can't decode byte 0xef
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 1622: ordinal not in range(128)
=> please run this command in your terminal
export LC_ALL=C.UTF-8
source ~/.bashrc
2.Resource punkt not found. / Resource wordnet not found.
please run this command in your terminal
python
>>> import nltk
>>> nltk.download('wordnet')
>>> nltk.download('punkt')