docs: get_num_tokens - default tokenizer

[nikitajz]

Hi!

1. Upd: Fixed in docs: update tokenizer notice in llms/getting_started #2641
   I've noticed in the Models -> LLM documentation the following note about get_num_tokens function:

Notice that by default the tokens are estimated using a HuggingFace tokenizer.

It looks not exactly correct since Huggingface is used only in legacy versions (< 3.8), so it probably outdated.

2. There is also a mapping in tiktoken package that can be reused in the function get_num_tokens:
   https://github.com/openai/tiktoken/blob/46287bfa493f8ccca4d927386d7ea9cc20487525/tiktoken/model.py#L13-L53

[AlexTs10]

I took a look at the repo and here the wrapper for openai uses the tiktoken library. If this doesn't help could you provide the exact files you saw this since you doc link doesn't show the exact code ?

[nikitajz]

I took a look at the repo and here the wrapper for openai uses the tiktoken library. If this doesn't help could you provide the exact files you saw this since you doc link doesn't show the exact code ?

I've added a PR with a tiny fix #2641
Hopefully, everything is correct.

[dosubot]

Hi, @nikitajz! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue was about a documentation error regarding the default tokenizer used in the get_num_tokens function. You mentioned that the note about HuggingFace tokenizer being the default is outdated and provided a link to the correct mapping in the tiktoken package. AlexTs10 confirmed that the wrapper for openai uses the tiktoken library, which is the correct mapping for the default tokenizer in the get_num_tokens function. You also added a pull request with a small fix to address the documentation error.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution and please don't hesitate to reach out if you have any further questions or concerns!