Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: cl100k_base tokenizer #2334

Closed
anthonypuppo opened this issue Aug 5, 2023 · 2 comments · Fixed by #2809
Closed

.Net: cl100k_base tokenizer #2334

anthonypuppo opened this issue Aug 5, 2023 · 2 comments · Fixed by #2809
Assignees
Labels
.NET Issue or Pull requests regarding .NET code

Comments

@anthonypuppo
Copy link
Contributor

The existing GPT3Tokenizer class uses p50k_base encoding. GPT-3.5, GPT-4, and text-embedding-ada-002 all rely on cl100k_base encoding.

The below table is extracted from the OpenAI notebook on token counting.

Encoding name OpenAI models
cl100k_base gpt-4, gpt-3.5-turbo, text-embedding-ada-002
p50k_base Codex models, text-davinci-002, text-davinci-003
r50k_base (or gpt2) GPT-3 models like davinci
@shawncal shawncal added .NET Issue or Pull requests regarding .NET code triage labels Aug 5, 2023
@anthonypuppo
Copy link
Contributor Author

SharpToken

@nacharya1 nacharya1 removed the triage label Aug 10, 2023
@nacharya1
Copy link
Contributor

@dluc are you able to look into this?

github-merge-queue bot pushed a commit that referenced this issue Sep 14, 2023
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Closes #2679
Closes #2508
Closes #2334

This PR contains changes to remove `GPT3Tokenizer` from Semantic Kernel
repository.

Alternatives:
https://github.com/microsoft/Tokenizer
https://github.com/dmitry-brazhenko/sharptoken

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Removed tokenizer logic, tests and related files.
2. Updated documentation.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
SOE-YoungS pushed a commit to SOE-YoungS/semantic-kernel that referenced this issue Nov 1, 2023
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Closes microsoft#2679
Closes microsoft#2508
Closes microsoft#2334

This PR contains changes to remove `GPT3Tokenizer` from Semantic Kernel
repository.

Alternatives:
https://github.com/microsoft/Tokenizer
https://github.com/dmitry-brazhenko/sharptoken

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Removed tokenizer logic, tests and related files.
2. Updated documentation.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
.NET Issue or Pull requests regarding .NET code
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants