the added special tokens seem to convert to ids uncorrectly with my chinese datasets

[AndDoIt]

Have you tried the argument of add_new_tokens? I used bert-base-chinese to train my chinese dataset, and the number of entity types exceeds the total [unused], after applying add_marker_tokens function, the vocab size and tokenizer length both added from 21128 to 21300. Then, I applied the origin BERT model, and applied resize_token_embeddings(), however, I found that the special token, such as <subj_start=地点>, <obj_start=人物>, etc, are all represented as 100, namely [UNK] in the sequence of input_ids after applying tokenizer.convert_tokens_to_ids(). Since the different marker tokens should have different id according to the paper, could you please help me to solve this problem?

[a3616001]

Yes, our ALBERT experiments are based on add_new_tokens. Did you try just to use tokenizer.tokenize()?

[AndDoIt]

Thanks for your reply!
I used tokenizer.tokenize() to find that the added_tokens.json did not work, it still only use the origin vocab.txt. What I really want is to keep the added token, rather than tokenized with wordPiece. Thus, I wondered whether re-pretraining is necessary so that I can use both origin and added tokens to apply the convert_tokens_to_ids() function in your experiment?

Could you please tell me the detailed steps of ALBERT experiments kindly, since I find the vocab_size was modified from 30000 to 30046 in rel-alb-ctx0.zip.

[a3616001]

Hi, I just want to check back and see whether you have resolved the issue.
I don't think you need to re-pretrain the model (we didn't in our ALBERT experiments). You may check this function to see how we augment the vocabulary.

[hero-png]

Hi, I also had this problem after trying to train a model with Bert base. Could you please answer me the following questions:

1. In the paper you also use base Bert to get the results of typed text markers. However, if I understand you correctly, Albert was used?

2. To add to that, in the code the "add_new_tokens" parameter is set to false per default and only to true for Albert. Why is that? If I understand the code correctly, only if this is set to true, the model will use the typed markers with entity labels and not the unknown token. Shouldn't this be true per default?

3. Can I just set the add_new_tokens parameter to true when I want to train base Bert (or any other model than albert) with it?

[a3616001]

@hero-png Thanks for your questions!

1. We tried both BERT and ALBERT as the base model in our experiments with typed markers.
2. For BERT experiments, we directly used the unused tokens in BERT vocabulary ([unused1], [unused2]) as the markers (check this line). For ALBERT experiments, we had to use add_new_tokens because there aren't pre-defined unused tokens in ALBERT vocabulary.
3. Yes, you can set add_new_tokens to be true when training BERT.

[hero-png]

@a3616001 Thank you very much for your quick reply!