Skip to content

aliuq/GPT-3-Encoder

 
 

Repository files navigation

GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

Forked from latitudegames/GPT-3-Encoder

Differences from the original repository:

  • By default, the encoder.json and vocab.bpe files are included in the package, which makes the file size very large.
  • It is possible to provide a CDN file address to load the files.

Usage

default

import { decode, encode } from '@aliuq/gpt-3-encoder'

encode(str)
decode(tokens)

fetch

encode/decode need an another parameter options, which is an object with the following properties:

  • encoder: the path of encoder.json file
  • bpe: the path of vocab.bpe file
import { decode, encode } from '@aliuq/gpt-3-encoder/fetch'

const options = {
  encoder: 'https://unpkg.com/@aliuq/gpt-3-encoder/dist/encoder.json',
  bpe: 'https://unpkg.com/@aliuq/gpt-3-encoder/dist/vocab.bpe',
}

encode(str, options)
decode(tokens, options)

You need to set moduleResolution in tsconfig.json to node16 or nodenext, otherwise it will prompt that the type declaration is not found, or you can use // @ts-expect-error to ignore the error, If fetch and globalThis.fetch do not exist, you need to install axios.

Also, you can copy the files to your own server and use them.

About

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • TypeScript 99.3%
  • JavaScript 0.7%