[Bug] Tokenizing with Llama 2 model freezes

[fozziethebeat]

Describe the bug
A clear and concise description of what the bug is.

I manually downloaded the new meta-llama/Llama-2-7b-hf model to my local disk and loaded it up with transformer.js tokenizer. When trying to tokenize a very particular string, tokenization halts and spins forever. Checking top, the node process is doing something but it's not clear what. I've let this run for at least 10 minutes and it doesn't finish.

How to reproduce
Steps or a minimal working example to reproduce the behavior

const breaking_string = `They wrote the following:
Momentum in your online business comes from (crystal) clarity around who you serve and how you’re going to serve them. 🤔   Over on The Art of Paid Traffic podcast today, I’m pumped to share with you an interview with one of my Accelerator Group Coaching students, Trish Taylor. 🤗                                                                                           Trish is a stylist (and former celebrity stylist at that), coach and brand style expert for business women, teaching them to own their worth and show up in an authentic and meaningful way. 🙌🏼                                                        sI asked Trish to come on the show because she’s gone through what I think is a super relatable transformation in her business -- that of working with clients 1-on-1 to now scaling her business online and creating online courses and programs. 👩🏼🏼 YOU
<200d>💻
As you’ll hear, it wasn’t until she had clarity on who exactly she serves that she was able to clearly create offers that serve this audience that she’s so passionate about.  ♀️
You\'ll also hear how she’s using a quiz and challenge launch, why I don't think she should count out webinars, the metrics she should be paying attention to when evaluating her ad campaigns, and so much more. 📊
Follow the link in my bio to give today's episode a listen and I’d love to hear from you…  are there any transformations YOU
 RE wanting to make in your business? 👇🏼                                                                                  *
. . . . . . . . .`;

export default async (argv: any) => {
  const { env, AutoTokenizer, PreTrainedTokenizer } = await eval(
    "import('@xenova/transformers')"
  );
  if (process.env.LOCAL_TRANSFORMER_PATH) {
    env.localModelPath = process.env.LOCAL_TRANSFORMER_PATH;
  }
  const pretrained = await AutoTokenizer.from_pretrained(
    "meta-llama/Llama-2-7b-hf"
  );
  const tokens = pretrained.encode(breaking_string);                                                                          console.log(tokens.length);
};

Pre-steps requires downloading the model locally and setting LOCAL_TRANSFORMER_PATH.

Expected behavior
A clear and concise description of what you expected to happen.

The string should be tokenized. The same string can be tokenized with mosaicml/mpt-7b-8k.

Logs/screenshots
If applicable, add logs/screenshots to help explain your problem.

Environment

• Transformers.js version: ^2.4.1
• Browser (if applicable):
• Operating system (if applicable): MacOS
• Other: Node v18.16.0

Additional context
Add any other context about the problem here.

[xenova]

I will test once I gain access to the llama-v2 models. In the meantime, I tested with the llama-v1 tokenizer and it works as intended, so it is most likely due to some change between tokenizer versions.

[fozziethebeat]

I'm going to see what debugging I can do as well.

I managed to narrow down the breaking string to something much smaller:

const breaking_string = `you…  `;

[xenova]

Thanks! By the way, can you confirm that the repo you're testing with contains the required files: tokenizer.json and tokenizer_config.json?

I've tested with a version on the hub with those files and it seems to work just fine?

<script type="module">
    import { AutoTokenizer } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.4.1';
let text = `you…  `


const pretrained = await AutoTokenizer.from_pretrained(
    "daryl149/llama-2-7b-chat-hf"
  );
const tokens = pretrained.encode(text);       
console.log(JSON.stringify(tokens));
// "[1,366,30098,259]"
</script>

JSFiddle demo: https://jsfiddle.net/jep6wky5/2/

[fozziethebeat]

I confirmed this.

I downloaded the model by doing

from huggingface_hub import snapshot_download
snapshot_download("meta-llama/Llama-2-7b-hf", local_dir="/Users/fozziethebeat/devel/models/meta-llama/Llama-2-7b-hf")

And manually verified there's both tokenizer.json and tokenizer_config.json. Both look normal to me and on spot checking seem to match what's on the hub.

Let me try the chat tokenizer. it could be the 7b tokenizer has an issue

[fozziethebeat]

I checked meta-llama/Llama-2-7b-chat-hf and it had the same problem. It's possible that downloading it locally is causing a problem but not sure how to rule that out just yet.

[xenova]

I have now got access to the models, but everything is working correctly for me. Can you confirm that transformers.js is loading models from the correct folder?

import { env } from '@xenova/transformers';
env.localModelPath = '/path/to/local/models/';

See docs for more information.

[fozziethebeat]

I've ruled out a few possible explanations. In my first runs I was running a typescript command that was being compiled by our tsconfig.

I've simplified the breaking example down to a simple mjs file:

import { env, AutoTokenizer, PreTrainedTokenizer } from "@xenova/transformers";

(async () => {
  const breaking_string = `you…  `;
  env.localModelPath = "/Users/fozziethebeat/devel/models/";
  const pretrained = await AutoTokenizer.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf"
  );
  const tokens = pretrained.encode(breaking_string);
  console.log(tokens.length);
})();

This is still breaking for me even without typescript and even with setting the local path manually.

[xenova]

Does this demo work in your browser: https://jsfiddle.net/jep6wky5/2/ ? It might be an issue with mac + node.js

[fozziethebeat]

That fiddle works for me. So it's likely something with running on mac & node.js.

[xenova]

I unfortunately do not have a mac to test with, so it's a little difficult to help debug. Are you certain it freezes at .encode()? Does this happen with other llama tokenizers like hf-internal-testing/llama-tokenizer? Are there no error logs?

If you are able to, I would appreciate if you cloned the repo and tested from source. In that way, you can trace through the code to see exactly where it freezes.

[fozziethebeat]

Yes that's my next step. I just woke up in my timezone so I'll start debugging more deeply.

[fozziethebeat]

Okay, I've figured out some things. I think it has something to do with text normalization, which for sure happens different in node and in the browser.

I found out that by manually changing tokenizer.json to have a pre_tokenizer such as

  "pre_tokenizer": {
    "type": "ByteLevel",
    "add_prefix_space": false,
    "trim_offsets": true,
    "use_regex": true
  },

(This is pulled from Mosaic's configs)

The problem goes away entirely. The tokenization is ugly but things work out.

I also found that by trimming away the trailing whitespace also happens to fix the problem. Apparently the … is some atrocious character that gets handled differently on some platforms and without a pre tokenizer, it passes through and may or may not get handled right.

The problem I diagnosed pretty deeply is that the while loop in the bpe method goes into an infinite loop. The word and pair variables keep getting the same value because the trailing whitespace remains and the ... value never gets found since it's not in the vocab.

[fozziethebeat]

Inspecting some other tokenization results, this isn't viable. it corrupts most strings. Need to figure out another fix.

[xenova]

Wow that's really strange?! 👀 Especially considering that it works in the browser (Mac/windows/Linux) as well as in node windows/Linux, but not for node mac.

Just out of curiosity, if you try the same with the python library, does it work correctly?

I don't want to speak too soon, but you might have found a bug with the mac version of node.js 👀. If you're able to test later versions of node.js, that could also be very useful! Thanks for investigating this!