Skip to content
This repository has been archived by the owner on Apr 23, 2024. It is now read-only.

Support custom tokens #97

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Support custom tokens #97

wants to merge 7 commits into from

Commits on Sep 14, 2022

  1. WIP: add special token

    @9173860
    9173860 committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    52c9397 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2022

  1. finished encode bpe with custom token

    @9173860
    9173860 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    7601884 View commit details
    Browse the repository at this point in the history

  2. support custom special token for encode

    @9173860
    9173860 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    b53046e View commit details
    Browse the repository at this point in the history

  3. fix vocab cli

    @9173860
    9173860 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    2197896 View commit details
    Browse the repository at this point in the history

  4. fix missed encoding custom token in certain circumstances

    @9173860
    9173860 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    20c848b View commit details
    Browse the repository at this point in the history

  5. add cli configs

    @9173860
    9173860 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    bd3ae7d View commit details
    Browse the repository at this point in the history

  6. update README.md with custom tokens feature

    @9173860
    9173860 committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    f4ac18d View commit details
    Browse the repository at this point in the history