Python: Add support for using other tokenizers #6217
Labels
enhancement
.NET
Issue or Pull requests regarding .NET code
python
Pull requests for the Python Semantic Kernel
Hi,
As part of developing a service-oriented prompt, I find it nearly mandatory to have a way to count tokens prior to sending them to the model. This allows us to modify the input to meet the API or local model's maximum token limitation, given the initial length of the input and prompt. This gives us the opportunity to prepare the code for forthcoming errors that are unwanted and can be controlled. I know that in Python, the Hugging Face tokenizer exists, but a similar solution seems to be lacking in C#.
Thanks for you time
The text was updated successfully, but these errors were encountered: