TokenTextSplitter ignores model_name when using from_tiktoken_encoder

[danb27]

System Info

langchain v0.0.162
python3.10

Who can help?

@hwchase17 I believe issue was introduced here: #2963

This works when calling __init__ directly, but the model_name is not passed to __init__ when using from_tiktoken_encoder()

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

print(TokenTextSplitter()._tokenizer)  # <Encoding 'gpt2'>
print(TokenTextSplitter(model_name="gpt-3.5-turbo")._tokenizer)  # <Encoding 'cl100k_base'>
print(TokenTextSplitter.from_tiktoken_encoder(model_name="gpt-3.5-turbo")._tokenizer)  # <Encoding 'gpt2'>

Expected behavior

print(TokenTextSplitter()._tokenizer)  # <Encoding 'gpt2'>
print(TokenTextSplitter(model_name="gpt-3.5-turbo")._tokenizer)  # <Encoding 'cl100k_base'>
print(TokenTextSplitter.from_tiktoken_encoder(model_name="gpt-3.5-turbo")._tokenizer)  # <Encoding 'cl100k_base'>