[Feature request] Support for `Prepend` type in `Normalizer.fromConfig`

[josephrocca]

Name of the feature
A user of OpenCharacters (where AutoTokenizer is used to support Basaran inference - not that this context is super relevant here) was trying to load this LLaMA model but hit upon the error "Unknown Normalizer type: Prepend"

This is what the config object looks like:

To replicate

await import("https://cdn.jsdelivr.net/npm/@xenova/transformers/dist/transformers.min.js");
let tokenizer = await AutoTokenizer.from_pretrained("https://huggingface.co/TehVenom/Pygmalion-7b-Merged-Safetensors/resolve/main");

[xenova]

Sure thing - I can add that 👍

[xenova]

Added in fa19890

It wasn't as simple as adding a prepend normalizer, since the llama tokenizer also does other things (like byte fallback)... but I got it working :)

Will be released in v2... (I promise, it will be released shortly haha 🤣 )

[josephrocca]

Thank you!!

[xenova]

Now in prerelease, which you can install the normal way.

npm i @xenova/transformers

You can then import and use as follows:

import { AutoTokenizer } from '@xenova/transformers';
let tokenizer = await AutoTokenizer.from_pretrained('TehVenom/Pygmalion-7b-Merged-Safetensors')
console.log(tokenizer('Hello world').input_ids)

Or you can import from a CDN:

<script type="module">
    import { AutoTokenizer } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers/dist/transformers.js';
</script>

Outputs a Tensor with the following data:

{
  data: [1, 15043, 3186],
  dims: [1, 3],
  size: 3,
  type: "int64"
}

---

This is much more similar to the standard HF from_pretrained syntax. For more info, see the docs.

Closing the issue for now :) Feel free to post any further questions.

[josephrocca]

Brilliant!

[xenova]

Happy to help :) It will take a little longer to get the llama models themselves running in the browser via ONNX, but feel free to suggest other tokenizers I can add.

[josephrocca]

@xenova Looks like there's some sort of minor bug related to pages served on localhost (unrelated to Normalizer stuff, but I figured it might not be worth its own issue). This code works fine on non-localhost tabs:

let { AutoTokenizer } = await import("https://cdn.jsdelivr.net/npm/@xenova/transformers@2.0.0-alpha.0/dist/transformers.js");
let tokenizer = await AutoTokenizer.from_pretrained("https://huggingface.co/TehVenom/Pygmalion-7b-Merged-Safetensors/resolve/main");

But if I run on a page that's served to localhost, then I get this error:

It seems to be adding https://huggingface.co to the start of the URL and also adding resolve/main, so it ends up requesting:

https://huggingface.co/https://huggingface.co/TehVenom/Pygmalion-7b-Merged-Safetensors/resolve/main/resolve/main/tokenizer_config.json

[xenova]

Yes - so, as I mentioned above, you must only pass in the repo id - not the full URL. This is different from v1.

let tokenizer = await AutoTokenizer.from_pretrained("TehVenom/Pygmalion-7b-Merged-Safetensors");

[josephrocca]

Ah, missed that - please forgive me and my feverish haste to implement this 😅

(I realise now that I was testing the above code in the browser console of huggingface.co, since I just happened to have that tab open, and the full URL version that I posted in my above message actually works there, somehow. So it's not a localhost vs non-localhost situation, it's huggingface.co vs non-huggingface.co that was causing my confusion)