Issue with Qwen 1.5 tokenizer

[mjp0]

System Info

transformers.js version 2.14.2.

Environment/Platform

• Website/web-app

Description

When I try to use tokenizers from https://huggingface.co/Qwen/Qwen1.5-14B-Chat with const t = new PreTrainedTokenizer(tok.tokenizer, tok.config).encode(text), I get the following error:

Uncaught (in promise) Error: SyntaxError: Invalid regular expression: /(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+/gu: Invalid group

I can't get anything more specific out from Chrome's debugger than that it happens with the PreTrainedTokenizer call.

All other tokenizers, like Mistral, Llama, etc. work perfectly, so I'm thinking that this must be a some sort of compatibility bug with Qwen.

Reproduction

1. Download https://huggingface.co/Qwen/Qwen1.5-14B-Chat/blob/main/tokenizer.json and https://huggingface.co/Qwen/Qwen1.5-14B-Chat/blob/main/tokenizer_config.json
2. Execute const t = new PreTrainedTokenizer(tokenizer, config).encode('foobar')

[xenova]

Hi there 👋 That is because we haven't yet merged the PR which adds support for Qwen2 models yet. PR should be merged shortly, and a release will follow

[mjp0]

Hi there 👋 That is because we haven't yet merged the PR which adds support for Qwen2 models yet. PR should be merged shortly, and a release will follow

Ah, that explains. I'll be eagerly waiting for the release to land, the new Qwen models are fantastic :)

[do-me]

Having a bit of a hard time with the Qwen tokenizer logic here too with `"@xenova/transformers": "^2.15.0",`, I always get: 

worker.js:112 Uncaught (in promise) DOMException: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': #\<Tensor> could not be cloned.  
   at GenerationConfig.callback\_function (webpack://semanticfinder/./src/js/worker.js?:112:30)  
   at Function.generate (webpack://semanticfinder/./node\_modules/@xenova/transformers/src/models.js?:1430:35)  
   at async Function.\_call (webpack://semanticfinder/./node\_modules/@xenova/transformers/src/pipelines.js?:929:32)  
   at async eval (webpack://semanticfinder/./src/js/worker.js?:105:34)  

Could you share your example for the live frontend updates for each beam generated by the model? I wasn't able to implement the logic here yet as it's somewhat different from the existing logic:

async function chat(text, max_new_tokens = 100) {
    return new Promise(async (resolve, reject) => {
        // hier Weiche einbauen für Qwen da tokenizer anders
        console.log(chat_model_name);

        if (chat_model_name.includes("Qwen")) {
            try {

                // Define the prompt and list of messages
                
                const messages = [
                    { "role": "system", "content": "You are a helpful assistant." },
                    { "role": "user", "content": text }
                ]

                const generatorText = chat_generator.tokenizer.apply_chat_template(messages, {
                    tokenize: false,
                    add_generation_prompt: true,
                });

                const thisChat = await chat_generator(generatorText, {
                    max_new_tokens: max_new_tokens,
                    do_sample: false,
                    callback_function: async function (beams) {
                        //const decodedText = await token_to_text(beams, chat_generator.tokenizer);
                        //console.log(decodedText);
                        console.log(beams)
                        self.postMessage({
                            type: 'chat',
                            chat_text: beams,
                        });

                        resolve(beams); // Resolve the main promise with chat text
                    },
                });
            } catch (error) {
                reject(error);
            }
        }

I guess some (new) parameters are just wrong.

Also, are you aware of the actual context length when using Qwen in transformers.js? I'm curious whether this might become
a bottleneck in the browser due to memory limitations.

[xenova]

#<Tensor> could not be cloned.

This is the main error, and occurs since you are trying to send the beams object (which contains Tensor objects) back to the main (UI) thread. I'd recommend decoding the output first, then sending that back. Something like chat_generator.tokenizer.decode(x[0].output_token_ids, { skip_special_tokens: true }) should work.

Also, beams is updated after every generation step, so you should only resolve once everything has returned.

[do-me]

x[0] that was the missing puzzle piece I was looking for, thank you so much!

It's working perfectly - just integrated it in SemanticFinder (here is the code if anyone wants to reuse).

The model is indeed remarkable for it's size and by far the best text generation model to be used with transformers.js atm.
Just a few questions:

1. are you aware of the actual context length when using Qwen in transformers.js?

2. Apparently, the max_new_token is not actually showing any effect if it's beneath a certain threshold. I tried to produce results with 10 or 50 tokens limit and they are always the same as the one produced with 100 tokens limit - just cut off in the middle of the sentence. Do you know anything more about this?

Side note: the model sometimes produces chinese signs in the middle of a sentence:

[xenova]

are you aware of the actual context length when using Qwen in transformers.js?

The config suggests 32k, but as you point out, I don't know what this will be in practice!

Apparently, the max_new_token is not actually showing any effect if it's beneath a certain threshold. I tried to produce results with 10 or 50 tokens limit and they are always the same as the one produced with 100 tokens limit - just cut off in the middle of the sentence. Do you know anything more about this?

This is the correct behaviour: max_new_tokens simply tells the model when to stop generating (does not affect what tokens are output)... this explains why it just chops off the end. Other parameters (like do_sample) control how generation works.

the model sometimes produces chinese signs in the middle of a sentence:

I believe this is an issue experienced by others too (not just transformers.js), and seems to be an issue with how the model was trained. Other than post-processing the logits to prevent generating chinese characters, I don't know how else this could be fixed without finetuning.

[do-me]

1. Very interesting, I'll run some tests and come back here with results.
2. I wasn't aware that stopping criteria are quite a philosophy per se. Will need to read a bit more about it...
3. That's what I expected too

[do-me]

I ran some tests by simply adding an information "Laura loves oranges" at the end of a text and then asking Qwen for this info.

A text with 5300 chars was processed fine in ~1 minute (i7, 32Gb RAM) and it gave the right answer even though with some hallucinated extra Laura likes orange because oranges are sweet

15000 chars were too much and the browser went out of memory:

I guess, there is not much to do about right now or do you think it's possible to overcome this limit somehow?

On a second thought, I am not sure whether maybe my computer went out of RAM but with peak 8Gb usage I doubt it. Could anyone with more RAM run some tests maybe?