Tokenization of input sentences for English

[polars05]

When feeding in sentences, do we need to pre-process them (e.g. using BPE) beforehand? Or do we just split them into words and let GPT-2 / BERT handle the pre-processing?

Example: If we want to use BERT embedding, do we have to pre-process "Jim Henson was a puppeteer" into ["jim", "henson", "was", "a", "puppet", "##eer"] using some external BPE library or do we just split the sentence on spaces and feed ["jim", "henson", "was", "a", "puppeteer"] into the model?

[BrikerMan]

You need to pre-process (tokenize) your sequence with the proper tool for the best performance. For example, if you are using the BERT embedding, use BERT Tokenizer for pre-processing.

If you just split the sentence with spaces and feed ["jim", "henson", "was", "a", "puppeteer"], It will lose some details. It will run thought, but not good as using BERT Tokenizer.

[polars05]

It seems that this part of the code already handles the tokenizing of the input sequence using the vocab.txt:

Kashgari/kashgari/embeddings/bert_embedding.py

Lines 87 to 98 in aeb1d78

	def _build_token2idx_from_bert(self):
	dict_path = os.path.join(self.model_folder, 'vocab.txt')
	token2idx = {}
	with codecs.open(dict_path, 'r', 'utf8') as reader:
	for line in reader:
	token = line.strip()
	token2idx[token] = len(token2idx)
	self.bert_token2idx = token2idx
	self.processor.token2idx = self.bert_token2idx
	self.processor.idx2token = dict([(value, key) for key, value in token2idx.items()])

Can I assume that we can just split the input sentence into ["jim", "henson", "was", "a", "puppeteer"] instead of ["jim", "henson", "was", "a", "puppet", "##eer"] before fitting it on a classifier built upon kashgari.embeddings.BERTEmbedding?

[BrikerMan]

Nope, We read the BERT vocab for token numerize process. It converts tokens to token-index number array. It handles the numerize process, not the tokenize process.
You can just split sentence by space, but you will get a little bit accurate feature representation when you tokenize it.

[polars05]

Ahh, thanks for the clarification! :) Guess would be helpful to state this in the documentation / include the tokenization process as part of the kashgari library

(Side note: the multi-label classification example shown in the documentation shows splitting of the sentences by whitespace; no tokenization via the BERT tokenizer was used as a preprocessing step)

[BrikerMan]

Yes, we should add those details. Maybe you could help me with the documents.

[polars05]

In the documentation (under "Quick Start") for the tf.keras branch, I added an example for pre-processing of text before feeding into BERT for text classification; I noticed kashgari builds off bert-keras so I've crafted the example based on the tokenizer from bert-keras.

I've created a pull request for that; hope the example helps with your documentation!

Side note: not sure if you have to set self.processor.add_bos_eos to False since the tokenizer will already automatically include the [CLS] and [SEP] tokens

Kashgari/kashgari/embeddings/bert_embedding.py

Line 80 in aeb1d78

self.processor.add_bos_eos = True

[BrikerMan]

In the documentation (under "Quick Start") for the tf.keras branch, I added an example for pre-processing of text before feeding into BERT for text classification; I noticed kashgari builds off bert-keras so I've crafted the example based on the tokenizer from bert-keras.

I've created a pull request for that; hope the example helps with your documentation!

Side note: not sure if you have to set self.processor.add_bos_eos to False since the tokenizer will already automatically include the [CLS] and [SEP] tokens

Kashgari/kashgari/embeddings/bert_embedding.py

Line 80 in aeb1d78

self.processor.add_bos_eos = True

Yes, if sequence includes the [CLS] and [SEP] tokens, we need to set add_bos_eos to False.

[BrikerMan]

@polars05 Added to the new document, check it out. https://kashgari.bmio.net/embeddings/bert-embedding/

[BrikerMan]

Added tokenizer property to the BERTEmebdding, simplified the process. Here is the document