Apparent bug for torchtext.data.vocab.Vocab causing collisions between OOV words and in vocab words.

[nryant]

When using torchtext version 0.3.1, I notice a rather alarming bug affecting torchtext.vocab.Vocab that causes collisions between OOV words and non-OOV words. When a vocabulary is instantiated with specials_first=True, there is a collision between OOV words and the first special word:

from collections import Counter
import torchtext
word_freqs = Counter({'the' : 10, 'apple' : 5, 'cat' : 2})
word_vocab = torchtext.vocab.Vocab(word_freqs, specials=['<pad>'], specials_first=True)
for word in ['<pad>', 'dfsddfsdfds']:
   print(word, word_vocab.stoi[word])

which yields:

('<pad>', 0)
('dfsddfsdfds', 0)

Similarly, when specials_first=False, there is a collision between OOV words and the highest frequency word in the vocabulary:

word_vocab = torchtext.vocab.Vocab(word_freqs, specials=['<pad>'], specials_first=False)
for word in ['the', 'dfsddfsdfds']:
    print(word, word_vocab.stoi[word])
('the', 0)
('dfsddfsdfds', 0)

The culprit appears to be the following line of vocab.py:

self.stoi.update({tok: i for i, tok in enumerate(self.itos)})

I assume this behavior is unintended. If so, it's a relatively easy fix assuming that _default_unk_index sill never be changed to output anything other than 0.

[mttk]

Yeah this is not the nicest bug. Assuming UNK=0 is ok since it's already hardcoded (so we can hardcode it further). I'll fix this when I get the time, if someone wants to take a stab at it, feel free.

[mttk]

Should be fixed via #482