Add Tokenizer Module (Pre-trained Tokenizer)

[gpengzhi]

The design of the pre-trained tokenizer module borrows ideas from both pytorch-transformer and our pretrained module.

To initiate a tokenizer, you can sepcify pretrained_model_name

tokenizer = BERTTokenizer(pretrained_model_name='bert-base-uncased')

or sepcify hparams

tokenizer = BERTTokenizer(hparams={'pretrained_model_name': 'bert-base-uncased'})

or directly load from the vocab file

tokenizer = BERTTokenizer(hparams={'pretrained_model_name': None, 'vocab_file': 'path_to_vocab'})

For the downstream tasks, you can add new tokens to the tokenizer

tokenizer.add_tokens([...])

and get the lasted vocabulary size, which can be used to define the downstream model

current_vocab_size = len(tokenizer)

You can also save the tokenizer to a directory (vocab file, special token file, added token file, and config file will be saved)

tokenizer.save('path-to-directory')

or load one from a directory

tokenizer = BERTTokenizer.load('path-to-directory')

Basically, we provide four core function for each tokenizer: text-to-token, text-to-id, id-to-text, token-to-text.

tokens = tokenizer(inputs=text, task='text-to-token')
ids = tokenizer(inputs=text, task='text-to-id')
tokens = tokenizer(inputs=ids, task='id-to-token')
text = tokenizer(inputs=ids, task='id-to-text')

[ZhitingHu]

Quick question: How would the tokenizers interact with the tx.data.Vocabulary class? Perhaps this is what to be considered when adding non-pretrained tokenizers (I remembered we discussed this in our weekly sync-up)?

[ZhitingHu]

Is the file test_sentencepiece.model intended? It has 247KB, make it smaller if possible

[gpengzhi]

Is the file test_sentencepiece.model intended? It has 247KB, make it smaller if possible

This is a standard file used in the testing of google/sentence. We can reuse it when we add SentencePieceTokenzier (that provides BPE, unigram language model functionality).

[huzecong]

@gpengzhi Regarding test_sentencepiece.model, can we download the file during testing? We do so for RecordData where we download a couple of images.

[gpengzhi]

@gpengzhi Regarding test_sentencepiece.model, can we download the file during testing? We do so for RecordData where we download a couple of images.

I think we can. I will change it accordingly.

[ZhitingHu]

Pls see the above unresolved comments

[ZhitingHu]

Looks good once the last comment on docstring is resolved.

[ZhitingHu]

Pls update bert, gpt2, xlnet examples to use these tokenizers, and delete unnecessary utils in those examples

[ZhitingHu]

Do the updated examples reproduce the previous results?

[gpengzhi]

Do the updated examples reproduce the previous results?

Yes

[ZhitingHu]

Once the method name is updated, it's good to merge