Skip to content

Latest commit

 

History

History
31 lines (21 loc) · 517 Bytes

models.mdx

File metadata and controls

31 lines (21 loc) · 517 Bytes
Raw

Models

## BPE

[[autodoc]] tokenizers.models.BPE

Model

[[autodoc]] tokenizers.models.Model

Unigram

[[autodoc]] tokenizers.models.Unigram

WordLevel

[[autodoc]] tokenizers.models.WordLevel

WordPiece

[[autodoc]] tokenizers.models.WordPiece The Rust API Reference is available directly on the Docs.rs website. The node API has not been documented yet.