You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After landing the WordPiece tokenizer (#22) and hammering out a layerized tokenizer design, we should also support for SentencePiece tokenization. SentencePiece is both language agnostic and reversible, and will be an important addition to our tokenization offering. SentencePiece graph ops are already supported by tensorflow text.
The text was updated successfully, but these errors were encountered:
After landing the WordPiece tokenizer (#22) and hammering out a layerized tokenizer design, we should also support for SentencePiece tokenization. SentencePiece is both language agnostic and reversible, and will be an important addition to our tokenization offering. SentencePiece graph ops are already supported by tensorflow text.
The text was updated successfully, but these errors were encountered: