Skip to content

Commit

Permalink
Add example
Browse files Browse the repository at this point in the history
  • Loading branch information
danny50610 committed Aug 19, 2023
1 parent 455aaf5 commit d03e116
Showing 1 changed file with 21 additions and 1 deletion.
22 changes: 21 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,27 @@
# bpe-tokeniser

![PHP Test](https://github.com/danny50610/bpe-tokeniser/actions/workflows/php.yml/badge.svg)
[![PHP Test](https://github.com/danny50610/bpe-tokeniser/actions/workflows/php.yml/badge.svg)](https://github.com/danny50610/php-cid/actions)
[![codecov](https://codecov.io/gh/danny50610/bpe-tokeniser/graph/badge.svg?token=CGORRQ1P6W)](https://codecov.io/gh/danny50610/bpe-tokeniser)
[![Latest Stable Version](http://poser.pugx.org/danny50610/php-cid/v)](https://packagist.org/packages/danny50610/php-cid)
[![Total Downloads](http://poser.pugx.org/danny50610/php-cid/downloads)](https://packagist.org/packages/danny50610/php-cid)

PHP port for [openai/tiktoken](https://github.com/openai/tiktoken) (most)

## Installation

```sh
composer require danny50610/bpe-tokeniser
```

## Example

### GPT-4 / GPT-3.5-Turbo (cl100k_base)
```php
<?php
use Danny50610\BpeTokeniser\EncodingFactory;

$enc = EncodingFactory::createByEncodingName('cl100k_base');

var_dump($enc->decode($enc->encode("hello world")));
// output: string(11) "hello world"
```

0 comments on commit d03e116

Please sign in to comment.