Kotok is a multiplatform library written in Kotlin. It is effectively a BPE (Byte Pair Encoding) tokeniser. It's developed to operate efficiently across varying platforms.
To include Kotok in your project, implement the following in your build.gradle file:
dependencies {
implementation "TODO"
}After installation, you can utilize Kotok as follows:
This project is licensed under the Apache 2.0 License.