Passing Tokenizer to LanguageModel

[yiyinglai]

Hello, I would like to use swift-transformers in my iOS application to simply development and avoid accessing network if possible.

I have downloaded a LLM model in coreML format and created LanguageModel with the loaded native coreML model. However, then I called generate on LanguageModel, it started to download something from the web and failed. I assume it is trying to download tokenizer, right?

Task <1F0DB5AB-3745-463B-AC02-D6D34023259A>.<1> finished with error [-1001] Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x3000c4570 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <1F0DB5AB-3745-463B-AC02-D6D34023259A>.<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <1F0DB5AB-3745-463B-AC02-D6D34023259A>.<1>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://huggingface.co/api/models/DeepSeek-R1-Distill-Qwen-1.5B-4bit/revision/main, NSErrorFailingURLKey=https://huggingface.co/api/models/DeepSeek-R1-Distill-Qwen-1.5B-4bit/revision/main, _kCFStreamErrorDomainKey=4}

I created AutoTokenizer from local tokenizer folder URL (under which tokenizer_config.json and tokenizer.json are available) successfully. Can I pass the created AutoTokenizer to LanguageModel?

[pcuenca]

Hi @yiyinglai,

Yes, it's most probably trying to download the tokenizer. However, the repository it's referring to is malformed, it should be of the form [org-name]/DeepSeek-R1-Distill-Qwen-1.5B-4bit instead of just DeepSeek-R1-Distill-Qwen-1.5B-4bit. I assume you are using private repos, but you need to make sure that the reference to the tokenizer includes both the namespace and the model name. I think the metadata entry in your Core ML model is not pointing to the right tokenizer.

If the tokenizer is available, it will only be downloaded on first use. In addition, if you download the tokenizer manually to the appropriate location, then it will be reused.

[yiyinglai]

Hi @pcuenca ,

I did create tokenizer from locally available files, but there is no way to pass it to the LanguageModel, right?

[pcuenca]

Currently there's not, unfortunately. We are in the process of revisiting those APIs, and one of our goals will be to better control and leverage cache layers. This information is very useful @yiyinglai! cc @mattt