Skip to content

Add HuggingFace tokenizer to executor runner #9455

New issue
New issue
Closed
Feature
Closed
Add HuggingFace tokenizer to executor runner#9455
Feature
Assignees
jackzhxng
Labels
module: llmIssues related to LLM examples and apps, and to the extensions/llm/ codetriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module
@jackzhxng

Description

@jackzhxng
jackzhxng
opened on Mar 20, 2025

🚀 The feature, motivation and pitch

Add .json tokenizer from https://github.com/pytorch-labs/tokenizers to the executor runner

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @larryliu0820 @mergennachin @cccclai @helunwencser

Metadata

Metadata

Assignees

Labels

module: llmIssues related to LLM examples and apps, and to the extensions/llm/ codetriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

Feature

Projects

ExecuTorch Core

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions