🧜🏻‍♂️ A tiny human language tokenizer framework in Swift.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Assets
Sample
Tokenizer
.gitignore
.swift-version
LICENSE
README.md
Tokenizer.podspec

README.md


Version Author Build Passing Swift
Platforms MIT
Cocoapods Carthage SPM

Introduction

What's this?

Tokenizer is a tiny human language tokenizer framework in Swift..

Preview

Preview

Installation

CocoaPods

pod 'Tokenizer'

Contribution

You are welcome to fork and submit pull requests.

License

Tokenizer is open-sourced software, licensed under the MIT license.

Usage

import Tokenizer

let stackoverflow = "Stack Overflow is the largest, most trusted online community for developers to learn, share their programming knowledge, and build their careers."
let baidu = "全球最大的中文搜索引擎、最大的中文网站。1999年底,身在美国硅谷的李彦宏看到了中国互联网及中文搜索引擎服务的巨大发展潜力,抱着技术改变世界的梦想,他毅然辞掉硅谷的高薪工作,携搜索引擎专利技术,于 2000年1月1日在中关村创建了百度公司。"

dump(stackoverflow.tokenize())
dump(Tokenizer.tokenize(text: baidu))