You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Code breaks using a different model other than BERT. I debugged into the code and found that the code is written with respect to BERT tokenizer only while the tokenizers of other transformer models are different. Below snippet in helpers.py
if BERT_TOKENIZER is None: # gets initialized during the first call to this method
if bert_pretrained_name_or_path:
BERT_TOKENIZER = transformers.BertTokenizer.from_pretrained(bert_pretrained_name_or_path)
BERT_TOKENIZER.do_basic_tokenize = True
BERT_TOKENIZER.tokenize_chinese_chars = False
else:
BERT_TOKENIZER = transformers.BertTokenizer.from_pretrained('bert-base-cased')
BERT_TOKENIZER.do_basic_tokenize = True
BERT_TOKENIZER.tokenize_chinese_chars = False
The text was updated successfully, but these errors were encountered:
Code breaks using a different model other than BERT. I debugged into the code and found that the code is written with respect to BERT tokenizer only while the tokenizers of other transformer models are different. Below snippet in helpers.py
if BERT_TOKENIZER is None: # gets initialized during the first call to this method
if bert_pretrained_name_or_path:
BERT_TOKENIZER = transformers.BertTokenizer.from_pretrained(bert_pretrained_name_or_path)
BERT_TOKENIZER.do_basic_tokenize = True
BERT_TOKENIZER.tokenize_chinese_chars = False
else:
BERT_TOKENIZER = transformers.BertTokenizer.from_pretrained('bert-base-cased')
BERT_TOKENIZER.do_basic_tokenize = True
BERT_TOKENIZER.tokenize_chinese_chars = False
Code breaks using a different model other than BERT. I debugged into the code and found that the code is written with respect to BERT tokenizer only while the tokenizers of other transformer models are different. Below snippet in helpers.py
The text was updated successfully, but these errors were encountered: