Skip to content

Commit

Permalink
Show list of models in README
Browse files Browse the repository at this point in the history
  • Loading branch information
simonw committed Jul 11, 2023
1 parent 286bd35 commit b1b0aae
Showing 1 changed file with 46 additions and 1 deletion.
47 changes: 46 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,55 @@ ttok Hello world --encode --tokens
[b'Hello', b' world']
```

## ttok --help
## Available models

This is the full list of available models and their corresponding encodings. Model names and encoding names are valid for the `-m/--model` option.

<!-- [[[cog
import cog
import tiktoken
output = []
for key, value in tiktoken.model.MODEL_TO_ENCODING.items():
output.append("- `{}` (`{}`)".format(key, value))
cog.out("\n".join(output))
]]] -->
- `gpt-4` (`cl100k_base`)
- `gpt-3.5-turbo` (`cl100k_base`)
- `text-davinci-003` (`p50k_base`)
- `text-davinci-002` (`p50k_base`)
- `text-davinci-001` (`r50k_base`)
- `text-curie-001` (`r50k_base`)
- `text-babbage-001` (`r50k_base`)
- `text-ada-001` (`r50k_base`)
- `davinci` (`r50k_base`)
- `curie` (`r50k_base`)
- `babbage` (`r50k_base`)
- `ada` (`r50k_base`)
- `code-davinci-002` (`p50k_base`)
- `code-davinci-001` (`p50k_base`)
- `code-cushman-002` (`p50k_base`)
- `code-cushman-001` (`p50k_base`)
- `davinci-codex` (`p50k_base`)
- `cushman-codex` (`p50k_base`)
- `text-davinci-edit-001` (`p50k_edit`)
- `code-davinci-edit-001` (`p50k_edit`)
- `text-embedding-ada-002` (`cl100k_base`)
- `text-similarity-davinci-001` (`r50k_base`)
- `text-similarity-curie-001` (`r50k_base`)
- `text-similarity-babbage-001` (`r50k_base`)
- `text-similarity-ada-001` (`r50k_base`)
- `text-search-davinci-doc-001` (`r50k_base`)
- `text-search-curie-doc-001` (`r50k_base`)
- `text-search-babbage-doc-001` (`r50k_base`)
- `text-search-ada-doc-001` (`r50k_base`)
- `code-search-babbage-code-001` (`r50k_base`)
- `code-search-ada-code-001` (`r50k_base`)
- `gpt2` (`gpt2`)
<!-- [[[end]]] -->

## ttok --help

<!-- [[[cog
from ttok import cli
from click.testing import CliRunner
runner = CliRunner()
Expand Down

0 comments on commit b1b0aae

Please sign in to comment.