New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try harder to find additional stable token #5

Closed

abdurraheemali opened this issue Dec 16, 2022 · 1 comment

abdurraheemali commented Dec 16, 2022

tiktoken/src/lib.rs

Lines 289 to 291 in ab3688a

    
           // TODO: we should try harder to find additional stable tokens 
        
           // This would reduce the amount of retokenising when determining completions 
        
           // Refer to the logic in an older version of this file

hauntsaninja mentioned this issue

Tell Rust you're not going to change the map #4

Closed

Collaborator

hauntsaninja commented Dec 16, 2022

hauntsaninja closed this as not planned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment