How to extend tokens dictionary?

[kpe]

It would be nice if an existing sentencepiece model could be extended with additional extra special/[MARKUP] tokens.

While adding real language tokens might not be needed (or possible), adding additional atomic markup tokens to an existing model, should be possible right?

As an usage scenario - imagine you have a plain text pre-trained model, and you want to handle some HTML-like document, i.e. you need to add special markup/atomic tokens like - [<P>], [<TABLE>], etc.

This is currently not possible, right?

[taku910]

SentencePiece supports user-defined symbols. I think you can use this feature.

https://github.com/google/sentencepiece/blob/master/doc/special_symbols.md

[kpe]

Thank you, @taku910 for your reply! What I would like to do is to take a sentencepiece model and extend it with new user-defined symbols (i.e. without changing the piece-to-id mapping for the existing pieces or the tokenization behavior.)

A use case would be to fine-tune some of the official pre-trained ALBERT models on the Google's Natural Question task (which includes some HTML tags), i.e. I would like to convert an existing sentencepiece model to a sentencepiece model that can also handle those additional user-defined symbols (let's say [<P>] and [</P>]) but without training a new model, which would potentially result in a different sentencepiece model (as the original plain text data might not be available).

[SchenbergZY]

Thank you, @taku910 for your reply! What I would like to do is to take a sentencepiece model and extend it with new user-defined symbols (i.e. without changing the piece-to-id mapping for the existing pieces or the tokenization behavior.)

A use case would be to fine-tune some of the official pre-trained ALBERT models on the Google's Natural Question task (which includes some HTML tags), i.e. I would like to convert an existing sentencepiece model to a sentencepiece model that can also handle those additional user-defined symbols (let's say [<P>] and [</P>]) but without training a new model, which would potentially result in a different sentencepiece model (as the original plain text data might not be available).

I also have the same problem, anything new?

[SchenbergZY]

My question is: is there a possible method to add some customer tokens at the end of an existing model with all score 0?

[gyc913]

I have the same question with SchenbergZY and kpe.

[taku910]

You can rewrite the model file to add new user defined symbols. It is also possible to rewrite the id<->piece mapping.

#398

[s4sarath]

@SchenbergZY - Have you find a hack regarding this. If anything please do update, if you don't mind. Thanks. :-)

[taku910]

The proto is a DSL and found at
https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto

protoc --python_out=. sentencepiece_model.proto

[s4sarath]

My bad. I took sometime to figure it out. :-) .

…

On Sat, Dec 7, 2019, 9:53 AM Taku Kudo ***@***.***> wrote: The proto is a DSL and found at https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto protoc --python_out=. sentencepiece_model.proto — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#426?email_source=notifications&email_token=ACRE6KGT7F332R4H6IQIC33QXMQMPA5CNFSM4JPFRIZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGF5OSY#issuecomment-562812747>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6KDG4MVPC24UPHZWQ3LQXMQMPANCNFSM4JPFRIZQ> .

[ilaouirine]

Hello,
Is it possible to define a pattern with user-defined symbols that I don't want to tokenize? For example, I don't want to tokenize any words starting with a '$' symbol. I also want to understand how decoding works because currently, all the words I define with user-defined symbols are assigned the ID 0.
Thank you!

[hardtoname1002]

Hello,
I'm confused about the sentence: "Control symbols must be inserted outside of the SentencePiece segmentation." Could you please explain this in detail, Thanks!