Tokenizers provided through TorchSharp?

[GeorgeS2019]

This issue (reference the TorchSharp discussion) is created to lobby .NET community interested in either classic or deep NLP to evaluate the use case of integrating BlingFire and provided it through e.g TorchSharp.Text

[GeorgeS2019]

Update 14thOct2021

BlingFire already integrated into ONNX Extension Ops

• blingfire_sentencebreaker.hpp

I wonder what else (from BlingFire) can be integrated. Has or should BlingFire being integrated and available from the distributed ortcustomops.dll?

Exported function

• RegisterCustomOps
• AddExternalCustomOp

Still exploring => can BlingFire learn from the implemented tokenizer to improve .NET tokenizer experience?