Revtok integration: reversible fields and subwords

[jekbradbury]

Integrates revtok, a simple fully-reversible tokenizer that optionally supports subwords. Adds the ReversibleField.reverse API to go from a padded batch of token IDs to a list of detokenized sentences. Speed is not great right now but I'm working on an implementation in a fast language and will add an optional wrapper for that in the Python revtok when it's ready.

Added a first stab at a test (want to release revtok to pypi before I add it to requirements.txt and the test can actually run); will add more.

Usage example:

>>> from torchtext import data, datasets
>>> TEXT = data.SubwordField()
>>> LABEL = data.Field(sequential=False)
>>> train, dev, test = datasets.SNLI.splits(TEXT, LABEL)
>>> LABEL.build_vocab(train)
>>> TEXT.build_vocab(train, max_size=2000) # currently takes about 8 minutes
>>> train[1].hypothesis
['\ue302 a ', ' person ', ' is ', ' at ', ' a ', ' diner ', ' ordering ', ' an ', ' omelette ', '. ']
>>> TEXT.segment(train, dev, test) # currently takes about 6 minutes
>>> train[1].hypothesis
['\ue302 a ', ' person ', ' is ', ' at ', ' a ', ' di', 'ner ', ', ', ' ', 'or', 'd', 'ering ', ' an ', ' o', 'm', 'el', 'e', 't', 'te ', '. ']
>>> train_iter, dev_iter, test_iter = data.BucketIterator.splits((train, dev, test), batch_size=16)
>>> b = next(iter(train))
>>> TEXT.reverse(b.premise.data)
['In a park a kid is chasing pigeons and two men are walking', 'A basketball player is hanging onto the rim while the ball is in the basket', 'An old man is speaking in a brown fedora and blue jacket.', ...]

[bmccann]

I added some comments where I think the test breaks right now. -- (changed the test in ac4f794)

It looks like the counts in the SubwordVocab are handled differently for special tokens (no subtract). Why is that?

Can you add comments in the subword field and vocab with a short description of how the handling differs from normal fields and vocabs and why?

I'm reading through revotk right now, but until then I think once the test is working this looks like a nice addition. We should probably wait until after the most recent wave of vocab additions and improvements go through before merging though.

[jekbradbury]

Notes from CI:

• declare encoding on field.py for Python 2
• fix subword TREC test to use CPU device
• fix Vocab tests that explicitly provide specials to also include '<unk>' (and note as a breaking change that also closes Adding is_label option in Field to remove unknown label #121)

[jekbradbury]

I think I messed up somewhere in a rebase, so the push build is failing on Python 2 due to a failure to round-trip the capitalization thing even though the PR build is succeeding and the branch claims to be up to date with master...

[windweller]

I installed revtok by pip install revtok...maybe the version is not correct?
This error was thrown:

  File "text_classifier.py", line 298, in <module>
    TEXT = data.ReversibleField(sequential=True, tokenize=tokenizer, lower=True)
  File "/torchtext/data/field.py", line 324, in __init__
    super(ReversibleField, self).__init__(**kwargs)
  File "/torchtext/data/field.py", line 148, in __init__
    self.tokenize = get_tokenizer(tokenize)
  File "/torchtext/data/utils.py", line 32, in get_tokenizer
    import revtok
  File "/revtok/__init__.py", line 1, in <module>
    from vocab.vocab import Vocab, OutOfVocabularyException
ImportError: No module named vocab.vocab

Is the revtok on pip is not the correct version, how should we install it??

[jekbradbury]

Currently you need to install from source https://github.com/jekbradbury/revtok