Skip to content
Choose a tag to compare


In this release, we have revamped the library to provide a more comprehensive experience for users to do NLP modeling using TorchText and PyTorch.

  • Migrated datasets to build on top of TorchData DataPipes
  • Added support RoBERTa and XLM-RoBERTa pre-trained models
  • Added support for Scriptable tokenizers
  • Added support for composable transforms and functionals


TorchText has modernized its datasets by migrating from older-style Iterable Datasets to TorchData’s DataPipes. TorchData is a library that provides modular/composable primitives, allowing users to load and transform data in performant data pipelines. These DataPipes work out-of-the-box with PyTorch DataLoader and would enable new functionalities like auto-sharding. Users can now easily do data manipulation and pre-processing using user-defined functions and transformations in a functional style programming. Datasets backed by DataPipes also enable standard flow-control like batching, collation, shuffling and bucketizing. Collectively, DataPipes provides a comprehensive experience for data preprocessing and tensorization needs in a pythonic and flexible way for model training.

from functools import partial
import torchtext.functional as F
import torchtext.transforms as T
from torch.hub import[ load_state_dict_from_url](
from import DataLoader
from torchtext.datasets import SST2

# Tokenizer to split input text into tokens
encoder_json_path = ""
vocab_bpe_path = ""
tokenizer = T.GPT2BPETokenizer(encoder_json_path, vocab_bpe_path)
# vocabulary converting tokens to IDs
vocab_path = ""
vocab = T.VocabTransform([load_state_dict_from_url](
# Add BOS token to the beginning of sentence
add_bos = T.AddToken(token=0, begin=True)
# Add EOS token to the end of sentence
add_eos = T.AddToken(token=2, begin=False)

# Create SST2 dataset datapipe and apply pre-processing
batch_size = 32
train_dp = SST2(split="train")
train_dp = train_dp.batch(batch_size).rows2columnar(["text", "label"])
train_dp =, input_col="text", output_col="tokens")
train_dp =, max_seq_len=254), input_col="tokens")
train_dp =, input_col="tokens")
train_dp =, input_col="tokens")
train_dp =, input_col="tokens")
train_dp =, padding_value=1), input_col="tokens")
train_dp =, input_col="label")
# create DataLoader
dl = DataLoader(train_dp, batch_size=None)
batch = next(iter(dl))
model_input = batch["tokens"]
target = batch["label"]

TorchData is required in order to use these datasets. Please install following instructions at


We have added support for pre-trained RoBERTa and XLM-R models. The models are torchscriptable and hence can be employed for production use-cases. The modeling APIs let users attach custom task-specific heads with pre-trained encoders. The API also comes equipped with data pre-processing transforms to match the pre-trained weights and model configuration.

import torch, torchtext
from torchtext.functional import to_tensor
xlmr_base = torchtext.models.XLMR_BASE_ENCODER
model = xlmr_base.get_model()
transform = xlmr_base.transform()
input_batch = ["Hello world", "How are you!"]
model_input = to_tensor(transform(input_batch), padding_value=1)
output = model(model_input)
torch.Size([2, 6, 768])

# add classification head
import torch.nn as nn
class ClassificationHead(nn.Module):
    def __init__(self, input_dim, num_classes):
        self.output_layer = nn.Linear(input_dim, num_classes)

    def forward(self, features):
        #get features from cls token
        x = features[:, 0, :]
        return self.output_layer(x)

binary_classifier = xlmr_base.get_model(head=ClassificationHead(input_dim=768, num_classes=2)) 
output = binary_classifier(model_input)
torch.Size([2, 2])

Transforms and tokenizers

We have revamped our transforms to provide composable building blocks to do text pre-processing. They support both batched and non-batched inputs. Furthermore, we have added support for a number of commonly used tokenizers including SentencePiece, GPT-2 BPE and CLIP.

import torchtext.transforms as T
from torch.hub import load_state_dict_from_url

padding_idx = 1
bos_idx = 0
eos_idx = 2
max_seq_len = 256
xlmr_vocab_path = r""
xlmr_spm_model_path = r""

text_transform = T.Sequential(
    T.Truncate(max_seq_len - 2),
    T.AddToken(token=bos_idx, begin=True),
    T.AddToken(token=eos_idx, begin=False),

text_transform([“Hello World”, “How are you”])


We have added an end-2-end tutorial to perform SST-2 binary text classification with pre-trained XLM-R base architecture and demonstrates the usage of new datasets, transforms and models.

Backward Incompatible changes

We have removed the legacy folder in this release which provided access to legacy datasets and abstractions. For additional information, please refer to the corresponding github issue (#1422) and PR (#1437)

New Features


  • Add XLMR Base and Large pre-trained models and corresponding transformations (#1407)
  • Added option to specify whether to load pre-trained weights (#1424)
  • Added Option for freezing encoder weights (#1428)
  • Enable optional return of all states in transformer encoder (#1430)
  • Added support for RobertaModel to accept model configuration (#1431)
  • Allow inferred scaling in MultiheadSelfAttention for head_dim != 64 (#1432)
  • Added attention mask to transformer encoder modules (#1435)
  • Added builder method in Model Bundler to facilitate model creation with user-defined configuration and checkpoint (#1442)
  • Cleaned up Model API (#1452)
  • Fixed bool attention mask in transformer encoder (#1454)
  • Removed xlmr transform class and instead used sequential for model transforms composition (#1482)
  • Added support for pre-trained Roberta encoder for base and large architecture #1491

Transforms, Tokenizers, Ops

  • Added ToTensor and LabelToIndex transformations (#1415)
  • Added Truncate Transform (#1458)
  • Updated input annotation type to Any to support torch-scriptability during transform composability (#1453)
  • Added AddToken transform (#1463)
  • Added GPT-2 BPE pre-tokenizer operator leveraging re2 regex library (#1459)
  • Added Torchscriptable GPT-2 BPE Tokenizer for RoBERTa models (#1462)
  • Migrated GPT-2 BPE tokenizer logic to C++ (#1469)
  • fix optionality of default arg in to_tensor (#1475)
  • added scriptable sequential transform (#1481)
  • Removed optionality of dtype in ToTensor (#1492)
  • Fixed max sequence length for xlmr transform (#1495)
  • add max_tokens kwarg to vocab factory (#1525)
  • Refactor vocab factory method to accept special tokens as a keyword argument (#1436)
  • Implemented ClipTokenizer that builds on top of GPT2BPETokenizer (#1541)


Migration of datasets on top of datapipes

Newly added datasets


  • Fix split filter logic in AmazonReviewPolarity (#1505)
  • use os.path.join for consistency. #1506
  • Fixing dataset test failures due to incorrect caching mode in AG_NEWS (#1517)
  • Added caching for extraction datapipe for AmazonReviewPolarity (#1527)
  • Added caching for extraction datapipe for Yahoo (#1528)
  • Added caching for extraction datapipe for yelp full (#1529)
  • Added caching for extraction datapipe for yelp polarity (#1530)
  • Added caching for extraction datapipe for DBPedia (#1571)
  • Added caching for extraction datapipe for SogouNews and AmazonReviewFull (#1594)
  • Fixed issues with extraction caching (#1550, #1551, #1552)
  • Updating Conll2000Chunking dataset to be consistent with other datasets (#1590)
  • [BC-breaking] removed unnecessary split argument from datasets (#1591)



Revamp TorchText dataset testing to use mocked data


  • Fixed attention mask testing (#1439)
  • Fixed CircleCI download failures on windows for XLM-R unit tests (#1441)
  • Asses unit tests for testing model training (#1449)
  • Parameterized XLMR and Roberta model integration tests (#1496)
  • Removed redundant get asset functions from parameterized_utils file (#1501)
  • Parameterize jit and non-jit model integration tests (#1502)
  • fixed cache logic to work with datapipes (#1522)
  • Convert get_mock_dataset fn in AmazonReviewPolarity to be private (#1543)
  • Removing unused TEST_MODELS_PARAMETERIZED_ARGS constant from model test (#1544)
  • Removed real dataset caching and testing in favor of mocked dataset testing (#1587)
  • fixed platform-dependent expectation for Multi30k mocked test. (#1593)
  • Fixed Conll2000Chunking Test (#1595)
  • Updated IWSLT testing to start from compressed file (#1596)
  • Used unicode strings to test utf-8 handling for all non-IWSLT dataset tests. (#1599)
  • Parameterize tests for similar datasets (#1600)


  • non-distributed training example for SST-2 binary text classification data using XLM-Roberta model (#1468)


Dataset Documentation

  • Updated docs for text classification and language modeling datasets (#1603)
  • Updated docs for Machine Translation, Sequence Tagging, Question Answer, Unsupervised Learning datasets (#1597)
  • Updated docs for CC100 and SST2 (#1604)
  • Update sphinx version, added rst files for models, transforms and functionals (#1434)
  • Removed experimental documentation (#1457)
  • Fix links in README (#1461)
  • Added sphinx based tutorial for SST-2 binary classification task using XLM-R model (#1468)
  • pointed to docs instead of outdated rtd link (#1480)
  • Added documentation describing XLM-R, the datasets it was trained on, and relevant license information (#1497)
  • Fixed CI doc build (#1504)
  • Remove example using next(...) from README (#1516)


  • Hide symbols when building third party code (#1467)
  • Add .DS_Store files to gitignore (#1470)
  • Remove Python 3.6 support as it has reached EOL (#1484)
  • Added .gitattributes file to hide generated circleci files in PRs (#1485)
  • Switched to use FileOpener from FileLoader (#1488)
  • Update python_requires in to reflect support for non-EOL python versions (#1521)
  • Added auto-formatters (#1545)
  • fix typo in torchtext/vocab/ (#1565)
  • Formatted datasets and tests (#1601, #1602)
Choose a tag to compare

This is a minor release compatible with PyTorch 1.10.2.

There is no feature change in torchtext from 0.11.1. For the full feature of v0.11.1, please refer to the v0.11.1 release notes.

Choose a tag to compare

torchtext 0.11.0 Release Notes

This is a relatively lightweight release while we are working on revamping the library. Users are encouraged to check various developments on the main branch.


  • Refactored C++ codebase to fix clang-tidy warnings and using emplace_back for improving performance (#1327)
  • Updated sentecepience to v0.1.95 to make it compilable on M1 (#1336)
  • Up the priority of numpy array comparison in self.assertEqual (#1341)
  • Removed mentions of conda-forge as it is no longer necessary to build on python 3.9 (#1345)
  • Separated experimental tests to help remove them easily during release cycles (#1348)
  • Splitted the pybind and torchtbind registration in separate file and refactor Vocab modules to allow vocab to be used in pure C++ environment (#1352)
  • Changed the default root directory for downloaded datasets to avoid dirtying the working directory (# 1361)
  • Added method for logging module usage in fbcode (#1367)
  • Updated bug report file (#1377)
  • Renamed default branch to main (#1378)
  • Enabled torchtext extension work seamlessly between fbcode and open-source (#1382)
  • Migrated CircleCI docker image (#1393)


  • Fix tag build so so that adding a tag will trigger a documentation build-and-upload (#1332)
  • Minor doc-string fix in Multi30K dataset (#1351)
  • Fixed example in doc-string of get_vec_by_tokens (#1383)
  • Updated docs to point to main instead of deprecated master branch (#1387)
  • Changed various links to point to main instead of master branch (#1392)

Bug fix

  • Fixed benchmark code that compares performance of vocab (#1339)
  • Fixed text classification example broken due removal of experimental datasets (#1347)
  • Fixed issue in IMDB dataset that result in all samples being positive depending on directory path (#1354)
  • Fixed doc building (#1365)
Choose a tag to compare

This release depends on pytorch 1.9.1
No functional changes other than minor updates to CI rules.

Choose a tag to compare


In this release, we introduce a new Vocab module that replaces the current Vocab class. The new Vocab provides common functional APIs for NLP workflows. This module is backed by an efficient C++ implementation that reduces look-up time by up-to ~85% for batch look-up (refer to summary of #1248 and #1290 for further information on benchmarks), and provides support for TorchScript. We provide accompanying factory functions that can be used to build the Vocab object either through a python ordered dictionary or an Iterator that yields lists of tokens.

creating Vocab from text file

import io
from torchtext.vocab import build_vocab_from_iterator
# generator that yield list of tokens
def yield_tokens(file_path):
    with, encoding = 'utf-8') as f:
       for line in f:
           yield line.strip().split()
# get Vocab object
vocab_obj = build_vocab_from_iterator(yield_tokens(file_path), specials=["<unk>"])

creating Vocab through ordered dict

from torchtext.vocab import vocab
from collections import Counter, OrderedDict
counter = Counter(["a", "a", "b", "b", "b"])
sorted_by_freq_tuples = sorted(counter.items(), key=lambda x: x[1], reverse=True)
ordered_dict = OrderedDict(sorted_by_freq_tuples)
vocab_obj = vocab(ordered_dict)

common API usage

# look-up index

# batch look-up indices
# support forward API of PyTorch nn Modules

# batch look-up tokens

# set default index to return when token not found 
vocab_obj["out_of_vocabulary"] #prints 0

Backward Incompatible changes

  • We have retired the old Vocab class into the legacy folder (#1289) . Users relying on this class should be able to access it from torchtext.legacy. The Vocab module that replaces this class is not backward compatible. The most notable difference is that the Vectors object is not an attribute of new Vocab object. We recommend users to use the build_vocab_from_iterator factory function to construct the new Vocab module that provides similar initialization capabilities as the retired Vocab class.
# retired Vocab class 
from torchtext.legacy.vocab import Vocab as retired_vocab
from collections import Counter
tokens_list = ["a", "a", "b", "b", "b"]
counter = Counter(tokens_list)
vocab_obj = retired_vocab(counter, specials=["<unk>","<pad>"], specials_first=True)

# new Vocab Module
from torchtext.vocab import build_vocab_from_iterator
vocab_obj = build_vocab_from_iterator([tokens_list], specials=["<unk>","<pad>"], specials_first=True)
  • Removed legacy batch from package (#1307) that was kept around for backward compatibility reasons. Users can still access batch from package.

New Features

  • Introduced functional to convert Iterable-style to map-style datasets (#1299)
from torchtext.datasets import IMDB
from import to_map_style_dataset
train_iter = IMDB(split='train')
#convert iterator to map-style dataset
train_dataset = to_map_style_dataset(train_iter)
  • Introduced functional to filter raw wikipedia XML dumps (#1292)
from import filter_wikipedia_xml
from torchtext.datasets import EnWik9
data_iter = EnWik9(split='train')
# filter data according to
filter_data_iter = filter_wikipedia_xml(data_iter)
# Added datasets for
from torchtext.datasets import Multi30k
train_data, valid_data, test_data = Multi30k()
# prints following 
#('Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.\n',
# 'Two young, White males are outside near many bushes.\n')


  • Separated experimental and legacy tests into separate subfolders (#1285)
  • Stored md5 hash instead of raw text data for in-built datasets testing (#1261)
  • Cleaned up CircleCI cache handling and optimization of daily cache (#1236, #1238)
  • Fixed CircleCI caching issue when new dataset is added (#1314)
  • Organized datasets by names in root folder and moved common file reading functions into dataset_utils (#1233)
  • Added unit-test to verify raw datasets name property (#1234)
  • Fixed jinja2 environment autoescape to enable select extensions (#1277)
  • Added yaml.safe_load instead of yaml.load (#1278)
  • Added defusedxml to parse untrusted XML data (#1279)
  • Added CodeQL and Bandit security checks as GitHub Actions (#1266)
  • Added benchmark code to compare Vocab module with python dict for batch look-up time (#1290)


  • Fixing doc for nn modules (#1267)
  • Store artifacts of rendered docs so that rendered docs can be checked on each PR (#1288)
  • Add Google Analytics support (#1287)

Bug Fix

  • Fixed import issue in text classification example (#1256)
  • Fixed and re-organized data pipeline example (#1250)


  • used c10::string_view and fast-text dictionary inside C++ kernel of Vocab module (#1248)
Choose a tag to compare


This is a minor release following pytorch 1.8.1. Please refer to torchtext 0.9.0 release note for more details.

Choose a tag to compare


In this release, we’re updating torchtext’s datasets to be compatible with the PyTorch DataLoader, and deprecating torchtext’s own DataLoading abstractions. We have published a full review of the legacy code and the new datasets in pytorch/text #664. These new datasets are simple string-by-string iterators over the data, rather than the previously custom set of abstractions such as Field. The legacy Datasets and abstractions have been moved into a new legacy folder to ease the migration, and will remain there for two more releases. For guidance about migrating from the legacy abstractions to use modern PyTorch data utilities, please refer to our migration guide (link).

The following raw text datasets are available as the replacement of the legacy datasets. Those datasets are iterators which yield the raw text data line-by-line. To apply those datasets in the NLP workflows, please refer to the end-to-end tutorial for the text classification task (link).

  • Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9
  • Text classification: AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB
  • Sequence tagging: UDPOS, CoNLL2000Chunking
  • Translation: IWSLT2016, IWSLT2017
  • Question answer: SQuAD1, SQuAD2

We add Python 3.9 support in this release

Backwards Incompatible

The current users of the legacy code will experience BC breakage as we have retired the legacy code (#1172, #1181, #1183). The legacy components are placed in folder as follows:

  • ->
  • ->
  • ->
  • ->
  • ->
  • ->

This means, all features are still available, but within torchtext.legacy instead of torchtext.

Table 1: Summary of the legacy datasets and the replacements in 0.9.0 release

Category Legacy 0.9.0 release
Language Modeling torchtext.legacy.datasets.WikiText2 torchtext.datasets.WikiText2
  torchtext.legacy.datasets.WikiText103 torchtext.datasets.WikiText103
  torchtext.legacy.datasets.PennTreebank torchtext.datasets.PennTreebank
  torchtext.legacy.datasets.EnWik9 torchtext.datasets.EnWik9
Text Classification torchtext.legacy.datasets.AG_NEWS torchtext.datasets.AG_NEWS
  torchtext.legacy.datasets.SogouNews torchtext.datasets.SogouNews
  torchtext.legacy.datasets.DBpedia torchtext.datasets.DBpedia
  torchtext.legacy.datasets.YelpReviewPolarity torchtext.datasets.YelpReviewPolarity
  torchtext.legacy.datasets.YelpReviewFull torchtext.datasets.YelpReviewFull
  torchtext.legacy.datasets.YahooAnswers torchtext.datasets.YahooAnswers
  torchtext.legacy.datasets.AmazonReviewPolarity torchtext.datasets.AmazonReviewPolarity
  torchtext.legacy.datasets.AmazonReviewFull torchtext.datasets.AmazonReviewFull
  torchtext.legacy.datasets.IMDB torchtext.datasets.IMDB
  torchtext.legacy.datasets.SST deferred
  torchtext.legacy.datasets.TREC deferred
Sequence Tagging torchtext.legacy.datasets.UDPOS torchtext.datasets.UDPOS
  torchtext.legacy.datasets.CoNLL2000Chunking torchtext.datasets.CoNLL2000Chunking
Translation torchtext.legacy.datasets.WMT14 deferred
  torchtext.legacy.datasets.Multi30k deferred
  torchtext.legacy.datasets.IWSLT torchtext.datasets.IWSLT2016, torchtext.datasets.IWSLT2017
Natural Language Inference torchtext.legacy.datasets.XNLI deferred
  torchtext.legacy.datasets.SNLI deferred
  torchtext.legacy.datasets.MultiNLI deferred
Question Answer torchtext.legacy.datasets.BABI20 deferred


  • Enable importing metrics/utils/functional from (#1229)
  • Set up daily caching mechanism with Master job (#1219)
  • Reset the functions in as private (#1224)
  • Resolve the download folder for some raw datasets (#1213)
  • Store the hash of the extracted CoNLL2000Chunking files so the extraction step will be skipped if the extracted files are detected (#1204)
  • Fix the total number of lines in doc strings of the datasets (#1200)
  • Extend CI tests to cover all the datasets (#1197, #1201, #1171)
  • Document the number of lines in the dataset splits (#1196)
  • Add hashes to skip the slow extraction if the extracted files are available (#1195)
  • Use decorator to loop over the split argument in the datasets (#1194)
  • Remove offset option from torchtext.datasets, and move torchtext.datasets.common to (#1188, #1145)
  • Remove the step to clean up the cache in test_iwslt() (#1192)
  • Split IWSLT dataset into IWSLT2016 and IWSLT2017 dataset and re-organize the parameters in the constructors (#1191, #1209)
  • Move the prototype datasets in torchtext.experimental.datasets.raw folder to torchtext.datasets folder (#1182, #1202, #1207, #1211, #1212)
  • Add a decorator add_docstring_header() to generate docstring (#1185)
  • Add EnWiki9 dataset (#1184)
  • Avoid unnecessary downloads and extraction for some raw datasets, and add more logging (#1178)
  • Split raw datasets into individual files (#1156, #1173, #1174, #1175, #1176)
  • Extend the unittest coverage for all the raw datasets (#1157, #1149)
  • Define the relative path of the datasets in the download_from_url() func and skip unnecessary download if the downloaded files are detected (#1158, #1155)
  • Add MD5 and NUM_LINES as the meta information in the __init__ file of torchtext.datasets folder (#1155)
  • Standardize the text dataset doc strings and argument order. (#1151)
  • Report the “exceeds quota” error for the datasets using Google drive links (#1150)
  • Add support for the string-typed split values to the text datasets (#1147)
  • Re-name the argument from data_select to split in the dataset constructor (#1143)
  • Add Python 3.9 support across Linux, MacOS, and Windows platforms (#1139)
  • Switch to the new URL for the IWSLT dataset (#1115)
  • Extend the language shortcut in func with the full name when Spacy tokenizers are loaded (#1140)
  • Fix broken CI tests due to spacy 3.0 release (#1138)
  • Pass an embedding layer to the constructor of the BertModel class in the BERT example (#1135)
  • Fix test warnings by switching to assertEqual() in PyTorch TestCase class (#1086)
  • Improve CircleCI tests and conda package (#1128, #1121, #1120, #1106)
  • Simplify TorchScript registration by adopting TORCH_LIBRARY_FRAGMENT macro (#1102)

Bug Fixes

  • Fix the total number of returned lines in setup_iter() func in RawTextIterableDataset (#1142)


  • Add number of classes to doc strings for text classification data (#1230)
  • Remove Lato font for pytorch/text website (#1227)
  • Add the migration tutorial (#1203, #1216, #1222)
  • Remove the legacy examples on pytorch/text website (#1206)
  • Update README file for 0.9.0 release (#1198)
  • Add CI check to detect undocumented parameters (#1167)
  • Add a static text link for the package version in the doc website (#1161)
  • Fix sphinx warnings and turn warnings into errors (#1163)
  • Add the text datasets to torchtext website (#1153)
  • Add the constructor document for IMDB and SST datasets (#1118)
  • Fix typos in the README file (#1089)
  • Rename "Arguments" to "Args" in the doc strings (#1110)
  • Build docs and push to gh-pages on nightly basis (#1105, #1111, #1112)
Choose a tag to compare


Updated pinned PyTorch version to 1.7.1 and added Python 3.9 support.


  • Added Python 3.9 support #1088
  • Added certifi for Windows unittest envir #1077
  • Added setup version to pin torch dependency #1067


  • Updated docs strings for torchtext.nn.InProjContainer #1083
  • Updated the doc strings for torchtext.nn.MultiheadAttentionContainer #1057
Choose a tag to compare

This is a relatively light release while we are working on revamping the library. According to PyTorch feature classification changes, the new building blocks and datasets in the experimental folder are defined as Prototype and available in the nightly release only. Once the prototype building blocks are matured enough, we will release them together with all the relevant commits in a beta release. At the same time, users are encouraged to take a look at those building blocks and give us feedback. An easy way to send your feedback is to open an issue in pytorch/text repo or comment in Issue #664. For details regarding the revamp execution, see Issue #985.

The nightly packages are accessible via Pip and Conda for Windows, Mac, and Linux. For example, Linux users can install the nightly wheels with the following command.

pip install --pre torch torchtext -f

For more detail instructions, please refer to Install PyTorch. It should be noted that the new building blocks are still under development, and the APIs have not been solidified.

The stable release branch here includes a few feature improvements and documentation updates. Compiled against the PyTorch 1.7.0 release, the stable release packages are available via Pip and Conda for Windows, Linux, and Mac.


  • Updated the BERT pipeline to improve question-answer task score #950
  • Fixed the order of the datasets used in the BERT example #1040
  • Skipped requests.get in download_from_url function if path exists #922
  • Used Ninja to build extensions and disable C++11 ABI when necessary for libtorch compatibility. #931
  • Removed SentencePiece from file. SentencePiece source code is now being used as the third-party library in torchtext #1055
  • Improved CircleCI setting for better engineering
    • Switched PyTorch binary location for CI unittests #1044
    • Parameterized UPLOAD_CHANNEL #1037
    • Installed binaries for the CI test directly from the CPU channel #1025, #981
    • Added dataclasses to dependencies for environment.yml #964
    • Bumped Xcode workers to 9.4.1 #951
    • Disabled glove tests due to URL breakage #920
    • Used the specific channel for the CI tests #907


  • Added test and updated error message for load_sp_model function in #984
  • Updated the README file in BERT example #899
  • Updated the legacy retirement message #1047
  • Updated index page to include links to PyTorch libraries and describe feature classification #1048
  • Cleaned up the doc strings #1049
  • Fixed clang-format version to what PyTorch uses #1052
  • Added OSX environment variables to the README file #1054
  • Updated README file for the prototype in the nightly release #1050

Bug Fixes

  • Fixed the order of the datasets used in the BERT example #1040
Choose a tag to compare


With the continued progress of PyTorch, some code in torchtext grew out of date with the SOTA PyTorch modules (for example, torchscript). In 0.7.0 release, we’re taking big steps toward modernizing torchtext, and adding warning messages to these legacy components which will be retired in the October 0.8.0 release. We’re also introducing a host of new features, including:

  1. A generalized MultiheadAttentionContainer for flexible attention behavior
  2. Torchscript support for SentencePiece models
  3. An end-to-end BERT example pipeline, including pertained weights and a question answering fine-tuning example
  4. The SQuAD1 and SQuAD2 question answering datasets
  5. Windows support

Legacy code and issues

For a period of time (ending around June of 2019), torchtext lacked active maintenance and grew out of date with the present SOTA research and PyTorch features. We’ve committed to bringing the library fully up to date, and identified a few core issues:

  • Several components and functionals were unclear and difficult to adopt. For example, the Field class coupled tokenization, vocabularies, splitting, batching and sampling, padding, and numericalization all together, and was opaque and confusing to users. We determined that these components should be divided into separate orthogonal building blocks. For example, it was difficult to use HuggingFace's tokenizers with the Field class (issue #609). Modular pipeline components would allow a third party tokenizer to be swapped into the pipeline easily.
  • torchtext’s datasets were incompatible with DataLoader and Sampler in, or even duplicated that code (e.g., Basic inconsistencies confused users. For example, many struggled to fix the data order while using Iterator (issue #828), whereas with DataLoader, users can simply set shuffle=False to fix the data order.

We’ve addressed these issues in this release, and several legacy components are now ready to be retired:

  • (link)
  • (link)
  • (link)
  • (link)

In 0.7.0 release, we add deprecation warnings, and finally will retire them to the torchtext.legacy directory in 0.8.0 release on October.

New dataset abstraction

Since the 0.4.0 release, we’ve been working on a new common interface for the torchtext datasets (inheriting from to address the issues above, and completed it for this release. For standard usage, we’ve created a map-style dataset which materializes the text iterator. A default dataset processing pipeline, including tokenizer and vocabulary, is added to the map-style datasets to support one-command data loading.

from torchtext.experimental.datasets import AG_NEWS
train, test = AG_NEWS(ngrams=3)

For those who want more flexibility, the raw text is still available as a by simply inserting .raw into the module path as follows.

train, test = torchtext.experimental.datasets.raw.AG_NEWS()

Instead of maintaining Batch and Iterator func in torchtext, the new dataset abstraction is fully compatible with like below. collate_fn is used to process the data batch generated from DataLoader.

from import DataLoader
def collate_fn(batch):
    texts, labels = [], []
    for label, txt in batch:
    return texts, labels
dataloader = DataLoader(train, batch_size=8, collate_fn=collate_fn)
for idx, (texts, labels) in enumerate(dataloader):
    print(idx, texts, labels)

With the new datasets, we worked together with the OSS community to re-write the legacy datasets in torchtext. Here is a brief summary of the progress:

  • Word language modeling datasets (WikiText2, WikiText103, PennTreeBank) #661, #774
  • Text classification datasets (AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull) #701, #775, #776
  • Sentiment analysis dataset (IMDb) #651
  • Translation datasets (Multi30k, IWSLT, WMT14) #751, #821, #851
  • Question-answer datasets (SQuAD1, SQuAD2) #773
  • Sequence tagging datasets (UDPOS, CoNLL2000Chunking) #805

Those new datasets stay in torchtext.experimental.datasets directory. The old version of the datasets are still available in torchtext.datasets and the new datasets are opt-in. In 0.8.0 release, the old datasets will be moved to torchtext.legacy directory.

To learn how to apply the new dataset abstraction with DataLoader and SOTA PyTorch compatibilities (like Distributed Data Parallel), we created a full example to use the new torchtext datasets (WikiText103, SQuAD1, etc) to train a BERT model. A pretrained BERT model is generated from masked language task and next sentence task. Then, the model is fine-tuned for the question-answer task. The example is available in torchtext repo (here).

Backwards Incompatible Changes

  • Remove code specific to Python2 #732

New Features


  • Integrates pytorch-probot into the repo #877
  • Switch to pytorch TestCase for build-in dataset #822
  • Switch experimental ngrams_func to data.utils.ngrams_iterator #813
  • Create root directory automatically for download_from_url if not exists #797
  • Add shebang line to suppress the lint warning #787
  • Switch to CircleCI and improve torchtext CI tests #744, #766, #768, #777, #783, #784, #794, #800, #801, #803, #809, #832, #837, #881, #888
  • Put sacremoses tokenizer test back #782
  • Update installation directions #763, #764, #769, #795
  • Add CCI cache for test data #748
  • Disable travis tests except for RUN_FLAKE8 #747
  • Disable Travis tests of which equivalent run on CCI #746
  • Use 'cpu' instead of None for Iterator #745
  • Remove the allow to fail statement in travis test #743
  • Add explicit test results to text classification datasets #738


  • Bump nightlies to 0.8.0 #847
  • Update README.rst file #735, #817
  • Update the labels of docs in text classification datasets #734

Bug Fixes



Add deprecation warning to legacy code #863. The following legacy components are ready to be retired, including

  • (link)
  • (link)
  • (link)
  • (link)
  • torchtext.datasets (link)

In 0.7.0 release, we add deprecation warnings, and finally will retire them to the torchtext.legacy directory in the October 0.8.0 release.