Skip to content
A natural language modeling framework based on PyTorch
Branch: master
Clone or download
Akshat Shrivastava and facebook-github-bot Scheduler with Warmup (#1184)
Pull Request resolved: #1184

Current implementations of warmup in pytext either involve doing warmup and optionally inverse square root decay (TODO) or using polynomial decay (TODO). However, through my experiments, I notice for large batch training a warmup period is helpful on other schedulers as well, especially when trying to mimic results of small batch training on large batches.

This diff adds support for `SchedulerWithWarmup`, underneath it holds two schedulers, WarmupScheduler and any other scheduler. After `warmup_steps`, the scheduler will switch from warmup to the specified scheduler.

This allows something like Warmup with Expontential Decay.

Since the scheduler is built on top of the existing warmup scheduler, any new features that come to that scheduler, will directly be applicable here.

Sample Config

"SchedulerWithWarmup": {
  "warmup_scheduler": {
    "warmup_steps": 500
  "scheduler": {
    "ExponentialLR": {
      "gamma": 0.95

Reviewed By: ArmenAg

Differential Revision: D18838272

fbshipit-source-id: 1b1b107552f2f8f38ed8cc319b9b64096d0bc07c
Latest commit a56c761 Dec 6, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Hide RNNG JIT till PyTorch 1.2 is ready (#686) Jun 12, 2019
.github Adding templates and changelog (#79) Dec 12, 2018
demo update XLM-R OSS tutorial and add Google Colab link (#1168) Nov 26, 2019
pytext Scheduler with Warmup (#1184) Dec 6, 2019
tests move sentencepiece tokenizer into (#1133) Nov 13, 2019
.coveragerc Initial commit Dec 11, 2018
.gitignore Text lint all .gitignore files May 15, 2019 Release PyText v0.3.0 (#1174) Dec 3, 2019 Adopt Contributor Covenant Aug 30, 2019 Gets rid of all doc gen warnings for Python 3.7 (#188) Jan 8, 2019
LICENSE Initial commit Dec 11, 2018 Make tutorial more trivial (add git clone) (#1037) Oct 8, 2019
activation_venv Specify CFLAGS to install fairseq in MacOS (#1175) Dec 3, 2019
activation_venv.bat Refine doc index page (#102) Dec 14, 2018
docs_requirements.txt Adding Google SentencePiece as a Tokenizer (#1106) Nov 8, 2019
install_deps Specify CFLAGS to install fairseq in MacOS (#1175) Dec 3, 2019
install_deps.bat Back out "EASY [PyText] use pip 18.1 to allow circleci tests to run" (#… Feb 20, 2019
pytest.ini Initial commit Dec 11, 2018
readthedocs.yml Gets rid of all doc gen warnings for Python 3.7 (#188) Jan 8, 2019
requirements.txt Resolve dependency conflict by specifying python-dateutil==2.8.0 (#1176) Dec 3, 2019 Release PyText v0.3.0 (#1174) Dec 3, 2019



PyText is a deep-learning based NLP modeling framework built on PyTorch. PyText addresses the often-conflicting requirements of enabling rapid experimentation and of serving models at scale. It achieves this by providing simple and extensible interfaces and abstractions for model components, and by using PyTorch’s capabilities of exporting models for inference via the optimized Caffe2 execution engine. We are using PyText in Facebook to iterate quickly on new modeling ideas and then seamlessly ship them at scale.

Core PyText features:

Installing PyText

PyText requires Python 3.6.1 or above.

To get started on a Cloud VM, check out our guide.

Get the source code:

  $ git clone
  $ cd pytext

Create a virtualenv and install PyText:

  $ python3 -m venv pytext_venv
  $ source pytext_venv/bin/activate
  (pytext_venv) $ pip install pytext-nlp

Detailed instructions and more installation options can be found in our Documentation. If you encounter issues with missing dependencies during installation, please refer to OS Dependencies.

Train your first text classifier

For this first example, we'll train a CNN-based text-classifier that classifies text utterances, using the examples in tests/data/train_data_tiny.tsv. The data and configs files can be obtained either by cloning the repository or by downloading the files manually from GitHub.

  (pytext_venv) $ pytext train < demo/configs/docnn.json

By default, the model is created in /tmp/

Now you can export your model as a caffe2 net:

  (pytext_venv) $ pytext export < demo/configs/docnn.json

You can use the exported caffe2 model to predict the class of raw utterances like this:

  (pytext_venv) $ pytext --config-file demo/configs/docnn.json predict <<< '{"text": "create an alarm for 1:30 pm"}'

More examples and tutorials can be found in Full Documentation.

Join the community


PyText is BSD-licensed, as found in the LICENSE file.

You can’t perform that action at this time.