How to use customized tokenizer?

[cch1219]

Hello. I am writing this post because I have a question about loading the tokenizer model. I am trying to use a pre-trained tokenizer in a Swift environment. After training, how do I apply the byproduct .model and .vocab files so that I can use the tokenizer I trained in Swift while using the swift-transformer API? I would appreciate it if you could answer.

[johnmai-dev]

I have the same problem. I plan to port mlx-community/Baichuan-M1-14B-Instruct-8bit and mlx-community/Hunyuan-7B-Instruct-3bit, both of which are customized tokenizers. When I intend to port their customized tokenizers, I encounter some issues and am unsure if this is correct.

1. These two models are not suitable for inheriting the PreTrainedTokenizerModel protocol because tokenizerData is not provided.
   

   swift-transformers/Sources/Tokenizers/Tokenizer.swift

   Lines 98 to 100 in 79626a8

   	public protocol PreTrainedTokenizerModel: TokenizingModel {
   	init(tokenizerConfig: Config, tokenizerData: Config, addedTokens: [String: Int]) throws
   	}

2. If inheriting the Tokenizer protocol, I need to reimplement chat_template.
   

   swift-transformers/Sources/Tokenizers/Tokenizer.swift

   Lines 147 to 209 in 79626a8

   	public protocol Tokenizer {
   	func tokenize(text: String) -> [String]
   	/// Main entry point
   	func encode(text: String) -> [Int]
   	func encode(text: String, addSpecialTokens: Bool) -> [Int]
   	func callAsFunction(_ text: String, addSpecialTokens: Bool) -> [Int]
   	/// Decode
   	func decode(tokens: [Int]) -> String
   	func decode(tokens: [Int], skipSpecialTokens: Bool) -> String
   	func convertTokenToId(_ token: String) -> Int?
   	func convertTokensToIds(_ tokens: [String]) -> [Int?]
   	func convertIdToToken(_ id: Int) -> String?
   	func convertIdsToTokens(_ ids: [Int]) -> [String?]
   	var bosToken: String? { get }
   	var bosTokenId: Int? { get }
   	var eosToken: String? { get }
   	var eosTokenId: Int? { get }
   	var unknownToken: String? { get }
   	var unknownTokenId: Int? { get }
   	var hasChatTemplate: Bool { get }
   	/// The appropriate chat template is selected from the tokenizer config
   	func applyChatTemplate(messages: [Message]) throws -> [Int]
   	/// The appropriate chat template is selected from the tokenizer config
   	func applyChatTemplate(messages: [Message], tools: [ToolSpec]?) throws -> [Int]
   	/// The appropriate chat template is selected from the tokenizer config
   	func applyChatTemplate(messages: [Message], tools: [ToolSpec]?, additionalContext: [String: Any]?) throws -> [Int]
   	/// The chat template is provided as a string literal or specified by name
   	func applyChatTemplate(messages: [Message], chatTemplate: ChatTemplateArgument) throws -> [Int]
   	/// The chat template is provided as a string literal
   	func applyChatTemplate(messages: [Message], chatTemplate: String) throws -> [Int]
   	func applyChatTemplate(
   	messages: [Message],
   	// A chat template can optionally be provided or specified by name when several templates are included in the tokenizer config. Normally this is not necessary.
   	chatTemplate: ChatTemplateArgument?,
   	addGenerationPrompt: Bool,
   	truncation: Bool,
   	maxLength: Int?,
   	tools: [ToolSpec]?
   	) throws -> [Int]
   	func applyChatTemplate(
   	messages: [Message],
   	// A chat template can optionally be provided or specified by name when several templates are included in the tokenizer config. Normally this is not necessary.
   	chatTemplate: ChatTemplateArgument?,
   	addGenerationPrompt: Bool,
   	truncation: Bool,
   	maxLength: Int?,
   	tools: [ToolSpec]?,
   	additionalContext: [String: Any]?
   	) throws -> [Int]
   	}

3. It requires dependencies on Swift Sentencepiece 和 Swift tiktoken . Can these dependencies be included in swift-transformers?

4. Can you give me some better suggestions for porting?

I hope to get your reply! Thank you! ♥️ @pcuenca @FL33TW00D

[pcuenca]

I think the way to go here would be to convert the custom tokenizers to the "fast" format supported by the tokenizers library, which would be immediately compatible with swift-transformers, rather than porting each custom tokenizer and requiring additional dependencies.

For example, the original Llama 4 models use a tiktoken tokenizer. When those models are converted to transformers format, the tokenizer is adapted too. Just for illustration, here's how it's done in that case (the conversion script also contains lots of code for weight conversion which is not relevant to this discussion).

This can be tedious to test, but so is porting the custom tokenization code.

I can get started with one of those just to see what this approach would look like.

[johnmai-dev]

I think the way to go here would be to convert the custom tokenizers to the "fast" format supported by the tokenizers library, which would be immediately compatible with swift-transformers, rather than porting each custom tokenizer and requiring additional dependencies.

For example, the original Llama 4 models use a tiktoken tokenizer. When those models are converted to transformers format, the tokenizer is adapted too. Just for illustration, here's how it's done in that case (the conversion script also contains lots of code for weight conversion which is not relevant to this discussion).

This can be tedious to test, but so is porting the custom tokenization code.

I can get started with one of those just to see what this approach would look like.

@pcuenca Thank you for your reply and suggestions. I may need to first learn how to convert other tokenizers to the "fast" format. I'm worried about whether there might be cases where conversion is not possible, requiring the implementation of custom tokenization code.

[pcuenca]

Well, there is the precedent of hundreds of models supported. But yes, we need to test.

[pcuenca]

Hi @johnmai-dev, here's a converted version of the Hunyuan tokenizer. The conversion code, as well as some extensive verifications, are documented in the model card.

I ran this quick test for verification in swift-transformers:

class SimpleHunyuanTests: XCTestCase {
    func testEncodeDecode() async throws {
        guard let tokenizer = try await AutoTokenizer.from(pretrained: "pcuenq/Hunyuan-7B-Instruct-tokenizer") as? PreTrainedTokenizer else {
            XCTFail()
            return
        }

        let cases = [
            ("ड़", [5619, 94, 5619, 120]),
            ("你好，世界", [109419, 3922, 100561]),
        ]
        for (text, expected) in cases {
            let encoded = tokenizer.encode(text: text)
            XCTAssertEqual(encoded, expected)
            let decoded = tokenizer.decode(tokens: encoded, skipSpecialTokens: true)
            XCTAssertEqual(decoded, text)
        }
    }
}

We are currently looking at Baichuan, but there are some edge cases that still don't work.

[pcuenca]

And here's the converted Baichuan tokenizer. This was especially tricky because of a weird behaviour in the original tokenizer. Thanks to @xenova we could find a fix!

It should work out of the box with swift-transformers, just as the Hunyuan one, @johnmai-dev.

[pcuenca]

cc @Vaibhavs10, as we talked about this earlier.

[johnmai-dev]

Awesome, thank you very much @pcuenca. This is a great job that will further promote the porting of more LLMs to Swift.🍻

[pcuenca]

Looking forward to your contributions! 🚀

[pcuenca]

Some of these hints could be added to a doc or guide of some sort, as similar requests are relatively frequent for new models. cc @FL33TW00D @mattt, just for visibility.