Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

README.md


Version Author Build Passing Swift
Platforms MIT
Cocoapods Carthage SPM

Introduction

What's this?

Tokenizer is a tiny human language tokenizer framework in Swift..

Preview

Preview

Installation

CocoaPods

pod 'Tokenizer'

Contribution

You are welcome to fork and submit pull requests.

License

Tokenizer is open-sourced software, licensed under the MIT license.

Usage

import Tokenizer

let stackoverflow = "Stack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers."
let baidu = "全球最大的中文搜索引擎、最大的中文网站。1999年底,身在美国硅谷的李彦宏看到了中国互联网及中文搜索引擎服务的巨大发展潜力,抱着技术改变世界的梦想,他毅然辞掉硅谷的高薪工作,携搜索引擎专利技术,于 2000年1月1日在中关村创建了百度公司。"

dump(stackoverflow.tokenize())
dump(Tokenizer.tokenize(text: baidu))

About

🧜🏻‍♂️ A tiny human language tokenizer framework in Swift.

Topics

Resources

License

Packages

No packages published