Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Same token same color #8

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

IsaacBreen
Copy link

@IsaacBreen IsaacBreen commented Apr 24, 2023

This PR assigns token colour by ID rather than by position. This makes highlighting more consistent.

Before (note that the second "kindness" has a different colour from the first "kindness".):

Screenshot 2023-04-24 at 11 43 48 am

After:

Screenshot 2023-04-24 at 11 43 31 am

This colour assignment persists for the duration of the session (until the user closes or refreshes the page). The benefit of this is that a change to the beginning of the text doesn't throw off the colours in subsequent parts. The tradeoff is that token colours may change between sessions, which might seem confusing to users. But I think the more immediate in-session consistency is worth the loss of between-session consistency.

I also changed the break behaviour from break-all to break-word to stop confusing breaking of words/numbers across lines.

@vercel
Copy link

vercel bot commented Apr 24, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
tiktokenizer ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 24, 2023 3:49am

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant