Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 

PCPM

Presenting Corpus of Pretrained Models. Links to pretrained models in NLP and voice with training script.

With rapid progress in NLP it is becoming easier to bootstrap a machine learning project involving text. Instead of starting with a base code, one can now start with a base pretrained model and within a few iterations get SOTA performance. This repository is made with the view that pretrained models minimizes collective human effort and cost of resources, thus accelerating development in the field.

Models listed are curated for either pytorch or tensorflow because of their wide usage.

Note: pytorch-transofmers is an awesome library which can be used to quickly infer/fine-tune from many pre-trained models in NLP. The pre-trained models from those are not included here.

Contents

Text ML

Language Models

Name Link Trained On Training script
Transformer-xl https://github.com/kimiyoung/transformer-xl/tree/master/tf#obtain-and-evaluate-pretrained-sota-models enwik8, lm1b, wt103, text8 https://github.com/kimiyoung/transformer-xl
GPT-2 https://github.com/openai/gpt-2/blob/master/download_model.py webtext https://github.com/nshepperd/gpt-2/
Adaptive Inputs (fairseq) https://github.com/pytorch/fairseq/blob/master/examples/language_model/README.md#pre-trained-models lm1b https://github.com/pytorch/fairseq/blob/master/examples/language_model/README.md

Permutation lanugage modelling Based - XLNet

Name Link Trained On Training script
XLnet https://github.com/zihangdai/xlnet/#released-models booksCorpus+English Wikipedia+Giga5+ClueWeb 2012-B+Common Crawl https://github.com/zihangdai/xlnet/

Masked Language Modelling Based - Bert

Name Link Trained On Training script
RoBERTa https://github.com/pytorch/fairseq/tree/master/examples/roberta#pre-trained-models booksCorpus+CC-N EWS+OpenWebText+CommonCrawl-Stories https://github.com/huggingface/transformers
BERT https://github.com/google-research/bert/ booksCorpus+English Wikipedia https://github.com/huggingface/transformers
MT-DNN https://mrc.blob.core.windows.net/mt-dnn-model/mt_dnn_base.pt (https://github.com/namisan/mt-dnn/blob/master/download.sh) glue https://github.com/namisan/mt-dnn

Machine Translation

Name Link Trained On Training script
OpenNMT http://opennmt.net/Models-py/ (pytorch) http://opennmt.net/Models-tf/ (tensorflow) English-German https://github.com/OpenNMT/OpenNMT-py
Fairseq (multiple models) https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md#pre-trained-models WMT14 English-French, WMT16 English-German https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md

Sentiment

Name Link Trained On Training script
Nvidia sentiment-discovery https://github.com/NVIDIA/sentiment-discovery#pretrained-models SST, imdb, Semeval-2018-tweet-emotion https://github.com/NVIDIA/sentiment-discovery
MT-DNN Sentiment https://drive.google.com/open?id=1-ld8_WpdQVDjPeYhb3AK8XYLGlZEbs-l SST https://github.com/namisan/mt-dnn

Reading Comprehension

SQUAD 1.1

Rank Name Link Training script
49 BiDaf https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz https://github.com/allenai/allennlp

Summarization

Model for English summarization

Name Link Trained On Training script
OpenNMT http://opennmt.net/Models-py/ Gigaword standard https://github.com/OpenNMT/OpenNMT-py

Speech to Text

Name Link Trained On Training script
NeMo-quartznet https://ngc.nvidia.com/catalog/models/nvidia:quartznet15x5 librispeech,mozilla-common-voice https://github.com/NVIDIA/NeMo
OpenSeq2Seq-Jasper https://nvidia.github.io/OpenSeq2Seq/html/speech-recognition.html#models librispeech https://github.com/NVIDIA/OpenSeq2Seq
Espnet https://github.com/espnet/espnet#asr-results librispeech,Aishell,HKUST,TEDLIUM2 https://github.com/espnet/espnet
wav2letter++ https://talonvoice.com/research/ librispeech https://github.com/facebookresearch/wav2letter
Deepspeech2 pytorch https://github.com/SeanNaren/deepspeech.pytorch/issues/299#issuecomment-394658265 librispeech https://github.com/SeanNaren/deepspeech.pytorch
Deepspeech https://github.com/mozilla/DeepSpeech#getting-the-pre-trained-model mozilla-common-voice, librispeech, fisher, switchboard https://github.com/mozilla/DeepSpeech
speech-to-text-wavenet https://github.com/buriburisuri/speech-to-text-wavenet#pre-trained-models vctk https://github.com/buriburisuri/speech-to-text-wavenet
at16k https://github.com/at16k/at16k#download-models NA NA

Datasets

Datasets referenced in this document

Language Model data

Common crawl

http://commoncrawl.org/

enwik8

Wikipedia data dump (Large text compression benchmark) http://mattmahoney.net/dc/textdata.html

text8

Wikipedia cleaned text (Large text compression benchmark) http://mattmahoney.net/dc/textdata.html

lm1b

1 Billion Word Language Model Benchmark https://www.statmt.org/lm-benchmark/

wt103

Wikitext 103 https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/

webtext

Original dataset not released by the authors. An open source collection is available at https://skylion007.github.io/OpenWebTextCorpus/

English wikipedia

https://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia

BooksCorpus

https://yknzhu.wixsite.com/mbweb https://github.com/soskek/bookcorpus

Sentiment

SST

Stanford sentiment tree bank https://nlp.stanford.edu/sentiment/index.html. One of the Glue tasks.

IMDB

IMDB movie review dataset used for sentiment classification http://ai.stanford.edu/~amaas/data/sentiment

Semeval2018te

Semeval 2018 tweet emotion dataset https://competitions.codalab.org/competitions/17751

Glue

Glue is a collection of resources for benchmarking natural language systems. https://gluebenchmark.com/ Contains datasets on natural language inference, sentiment classification, paraphrase detection, similarity matching and lingusitc acceptability.

Speech to text data

fisher

https://pdfs.semanticscholar.org/a723/97679079439b075de815553c7b687ccfa886.pdf

librispeech

www.danielpovey.com/files/2015_icassp_librispeech.pdf

switchboard

https://ieeexplore.ieee.org/document/225858/

Mozilla common voice

https://github.com/mozilla/voice-web

vctk

https://datashare.is.ed.ac.uk/handle/10283/2651

Hall of Shame

High quality research which doesn't include pretrained models and/or code for public use.

Non English

Other Collections

Allen NLP

Built on pytorch, allen nlp has produced SOTA models and open sourced them. https://github.com/allenai/allennlp/blob/master/MODELS.md

They have neat interactive demo on various tasks at https://demo.allennlp.org/

GluonNLP

Based on MXNet this library has extensive list of pretrained models on various tasks in NLP. http://gluon-nlp.mxnet.io/master/index.html#model-zoo

About

Presenting Collection of Pretrained Models. Links to pretrained models in NLP and voice.

Topics

Resources

License

Releases

No releases published

Packages

No packages published