Joey NMT was originally meant for translation purposes. Our goal was to alleviate experimenting with speech2text, which is why we modified the original Codebase to accept and process audio data as source and text data as target transcriptions.
Check out the detailed documentation of the original project and their paper. Our paper can be found in this repository.
Joey-S2T is developed by Niklas Korz (Heidelberg University), Yoalli Rezepka García (Heidelberg University) and David Hector (Heidelberg University). Joey NMT is developed by Jasmijn Bastings (University of Amsterdam) and Julia Kreutzer (Heidelberg University).
- Speech2Text for all languages provided by the Commonvoice Dataset
Joey S2T is built on PyTorch, torchaudio and torchtext for Python >= 3.5.
A. From source
- Clone this repository:
git clone https://github.com/DavidHector/joeynmt.git - Install joeynmt and it's requirements:
cd joeynmtpip3 install .(you might want to add--userfor a local installation).
Warning! When running on GPU you need to manually install the suitable PyTorch version for your CUDA version. This is described in the PyTorch installation instructions.
Experiments are specified in configuration files, in simple YAML format. You can find an example in the configs directory.
Edit the language parameter in stt.yaml to run Speech2Text for different languages.
Most importantly, the configuration contains the description of the model architecture (e.g. number of hidden units in the encoder RNN) and the training hyperparameters (learning rate, validation frequency etc.).
For training, we recommend running our extension on Google Colab. Run:
python3 -m joeynmt train configs/stt.yaml.
This will train a model on the training data specified in the config (here: stt.yaml),
validate on validation data,
and store model parameters, vocabularies, validation outputs and a small number of attention plots in the model_dir (also specified in config).
Tip: Be careful not to overwrite models, set overwrite: False in the model configuration.
The validations.txt file in the model directory reports the validation results at every validation point.
Models are saved whenever a new best validation score is reached, in batch_no.ckpt, where batch_no is the number of batches the model has been trained on so far.
best.ckpt links to the checkpoint that has so far achieved the best validation score.
For training on a GPU, set use_cuda in the config file to True. This requires the installation of required CUDA libraries.
In order to transcribe the contents of an audio file, simply run
python3 -m joeynmt translate configs/stt.yaml --input_path < my_input.mp3 > .
The transcriptions will be written to stdout or alternatively--output_path if specified.
Here we'll collect projects and repositories that are based on Joey, so you can find inspiration and examples on how to modify and extend the code.
- Joey Toy Models. @bricksdont built a collection of scripts showing how to install JoeyNMT, preprocess data, train and evaluate models. This is a great starting point for anyone who wants to run systematic experiments, tends to forget python calls, or doesn't like to run notebook cells!
- African NMT. @jaderabbit started an initiative at the Indaba Deep Learning School 2019 to "put African NMT on the map". The goal is to build and collect NMT models for low-resource African languages. The Masakhane repository contains and explains all the code you need to train JoeyNMT and points to data sources. It also contains benchmark models and configurations that members of Masakhane have built for various African languages. Furthermore, you might be interested in joining the Masakhane community if you're generally interested in low-resource NLP/NMT.
- Slack Joey. Code to locally deploy a Joey NMT model as chat bot in a Slack workspace. It's a convenient way to probe your model without having to implement an API. And bad translations for chat messages can be very entertaining, too ;)
- Flask Joey. @kevindegila built a flask interface to Joey, so you can deploy your trained model in a web app and query it in the browser.
- User Study. We evaluated the code quality of this repository by testing the understanding of novices through quiz questions. Find the details in Section 3 of the Joey NMT paper.
- Self-Regulated Interactive Seq2Seq Learning. Julia Kreutzer and Stefan Riezler. Published at ACL 2019. Paper and Code. This project augments the standard fully-supervised learning regime by weak and self-supervision for a better trade-off of quality and supervision costs in interactive NMT.
- Speech Joey. @Sariyusha is giving Joey ears for speech translation. Code.
- Hieroglyph Translation. Joey NMT was used to translate hieroglyphs in this IWSLT 2019 paper by Philipp Wiesenbach and Stefan Riezler. They gave Joey NMT multi-tasking abilities.
If you use Joey NMT in a publication or thesis, please cite the following paper:
@inproceedings{kreutzer-etal-2019-joey,
title = "Joey {NMT}: A Minimalist {NMT} Toolkit for Novices",
author = "Kreutzer, Julia and
Bastings, Jasmijn and
Riezler, Stefan",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations",
month = nov,
year = "2019",
address = "Hong Kong, China",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/D19-3019",
doi = "10.18653/v1/D19-3019",
pages = "109--114",
}
Joeys are infant marsupials.