bpe

Here are 4 public repositories matching this topic...

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

✂️ OpenAI's tiktoken tokenizer written in Go

Golang BPE (Bytes Pair Encoding) algorithm implementation.

A BPE tokenizer built with golang

Add a description, image, and links to the bpe topic page so that developers can more easily learn about it.

To associate your repository with the bpe topic, visit your repo's landing page and select "manage topics."