v1.1.2

AbrahamSanders released this 26 Sep 07:54

· 23 commits to main since this release

a058f5f

Fix for exclusive upper bound on max_token_length parameter to BpeTrainer. This parameter is set when passing the --max_token_codebook_ngrams argument in codec_bpe.train_tokenizer

Assets 2