Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace GPT2TokenizerFast with tiktoken-based tokenizer (#44) #44

Closed
wants to merge 4 commits into from

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Apr 11, 2023

This PR fixes issue #43 by replacing GPT2TokenizerFast with a tiktoken-based tokenizer. The new implementation supports both messages for chat completions models and simple strings for ordinary completions models.

List of commits:

  1. Implement tiktoken-based tokenizer in autopr.utils.tokenizer
  2. Update tokenizer usage in autopr/utils/repo.py
  3. Update tokenizer usage in autopr/repos/completions_repo.py
  4. Add tests for the new tiktoken-based tokenizer

@github-actions github-actions bot changed the title Replace GPT2TokenizerFast with tiktoken Replace GPT2TokenizerFast with tiktoken and update function calls Apr 11, 2023
@github-actions github-actions bot changed the title Replace GPT2TokenizerFast with tiktoken and update function calls Replace GPT2TokenizerFast with tiktoken Apr 11, 2023
@github-actions github-actions bot changed the title Replace GPT2TokenizerFast with tiktoken Replace GPT2TokenizerFast with tiktoken-based tokenizer (#44) Apr 11, 2023
@irgolic irgolic closed this Nov 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant