Add `add_bos`, `add_eos` parameters to `SentencePieceTokenizer`.

[briango28]

Is your feature request related to a problem? Please describe.
SentencePieceTokenizer, which is a wrapper for tf_text.SentencepieceTokenizer, currently does not expose all the internal parameters that may be specified in tf_text.SentencepieceTokenizer.__init__().
The parameters I am particularly interested in are add_bos and add_eos; in the current state, users must explicitly add the token ids for '<s>' and '</s>' (which default to 1 and 2) to the result of SentencePieceTokenizer.tokenize().

Describe the solution you'd like
Add add_bos=False, add_eos=False to SentencePieceTokenizer.__init__(), save them in self.add_bos and self.add_eos, and use them in set_proto() when initializing tf_text.SentencepieceTokenizer.

Describe alternatives you've considered
It's always possible to write a custom Tokenizer by subclassing, but the change seemed trivial enough.

[mattdangerw]

The idea with KerasNLP tokenizers is to keep the basic functionality pretty pure. string -> ragged int and ragged int -> string. And then to add a layer like a StartEndPacker layer on top for bos, eos, padding, etc.

Is there a reason that doesn't work for your use case?

[mattdangerw]

It definitely might make sense to add the arguments here, as sentencepiece does have build in bos/eos tokens unlike other tokenizers. But would like to understand the overall use case first.

[briango28]

The primary use case I thought of was for porting preprocessing pipelines from other frameworks.
If a user wants to match the format of the original implementation, they must explicitly add a Packer layer after SentencePieceTokenizer unless the original implementation intentionally omitted the emission of bos & eos tokens.

While I feel this is perfectly fine in itself, I've found that having to manually load the sentencepiece model to copy bos/eos token ids or strings (in case of dtype='string') to the Packer layer is rather unwieldy.
If adding too many options to SentencePieceTokenizer.__init__() is undesirable, perhaps adding attributes to expose this information would be acceptible?

[mattdangerw]

Yeah, I think this makes sense, just cause the beginning and end tokens are actual special in the sentencepiece library and proto. Let's just add them to init. Would you be interested in making the change?

[briango28]

Yes, I would. I'll follow up with a PR in the near future.

[mattdangerw]

This is fixed!