[Bug] Unknown model class "xlm-roberta", attempting to construct from base class.

[vasileermicioi]

Describe the bug
A clear and concise description of what the bug is.

Unknown model class "xlm-roberta", attempting to construct from base class.
Malformed class definition. [class PreTrainedModel extends Callable]
Error: Unable to load model: Xenova/multilingual-e5-base. Please report this bug at https://github.com/xenova/transformers.js/issues/new/choose.
at Function.from_pretrained (file:///home/vasile/dev/experiments/transformers/node_modules/@xenova/transformers/src/models.js:657:19)
at Function.from_pretrained (file:///home/vasile/dev/experiments/transformers/node_modules/@xenova/transformers/src/models.js:2451:42)

[vasileermicioi]

 const extractor = await pipeline(
    "feature-extraction",
    "Xenova/multilingual-e5-base",
    { revision: "main" }
  );
  const result = await extractor("This is a simple test.", {
    pooling: "mean",
    normalize: true,
  });
  console.log(result);

[xenova]

Thanks for the report! Will get this fixed :)

[xenova]

This functionality has now been added 👍 (9a51c99)

const extractor = await pipeline(
    "feature-extraction",
    "Xenova/multilingual-e5-base",
    { revision: "main" }
);
const result = await extractor("This is a simple test.", {
    pooling: "mean",
    normalize: true,
});
console.log(result);
// Tensor {
//     data: Float32Array(768) [0.016873687505722046, 0.029895609244704247, ...],
//     dims: [1, 768],
//     size: 768,
//     type: "float32",
// }

... will publish a new release soon so that you can use it 👍

[xenova]

Now available in v2.3.1 (https://www.npmjs.com/package/@xenova/transformers) 👍

[vasileermicioi]

any chance to get the large one quantized?
it is only recently uploaded, see https://huggingface.co/intfloat/multilingual-e5-large

[xenova]

Done 👍 (I also added the small one)

• https://huggingface.co/Xenova/multilingual-e5-large
• https://huggingface.co/Xenova/multilingual-e5-small

Note that since the large model is >2GB, you'll have to use the quantized version for now (which may result in poorer performance). There's also a "medium" version: https://huggingface.co/Xenova/multilingual-e5-base you can test out.

[vasileermicioi]

There's also a "medium" version: https://huggingface.co/Xenova/multilingual-e5-base you can test out.
with this one I got the bug :)

[vasileermicioi]

tested the large one - works fine,
for Xenova/multilingual-e5-small I got

Error: An error occurred during model execution: "Missing the following inputs: token_type_ids.

[xenova]

tested the large one - works fine, for Xenova/multilingual-e5-small I got

Error: An error occurred during model execution: "Missing the following inputs: token_type_ids.

Hmm, looks like - for some reason - the author is mixing tokenizers and models. As seen in the config file:

{
  "architectures": [
    "BertModel"
  ],
  "tokenizer_class": "XLMRobertaTokenizer",
}

So, just as a workaround for now, you can override the call function of the tokenizer to add token type ids like this:

const extractor = await pipeline(
    "feature-extraction",
    "Xenova/multilingual-e5-small",
    { revision: "main" }
);

extractor.tokenizer.prepare_model_inputs = (model_inputs) => {
    const token_type_ids = model_inputs.attention_mask.clone();
    token_type_ids.data.fill(0n);
    return {...model_inputs, token_type_ids};
};
const result = await extractor("This is a simple test.", {
    pooling: "mean",
    normalize: true,
});
console.log(result);
// Tensor {
//   data: Float32Array(384) [0.019251275807619095, 0.008903810754418373, -0.05832963064312935, ...]
//   dims: [1, 384],
//   size: 384,
//   type: "float32"
// }

---

It might be worth checking the model's input parameters and if they are missing, just use the default (all 0's).

[vasileermicioi]

the workaround doesn't work, I am getting NaNs for embeddings

but also the quality as described in the card is much lower than the base one compared to base to large
e.g. first line on their evaluation "accuracy on MTEB AmazonCounterfactualClassification (en)"
73.791(small) - 78.970(base) - 79.060(large)

the diff between base and large is really small, but small one is loosing constantly ~5points on every test

so I would expect the avg in the leaderboard to decrease by about 5 points for the small one compared to base
https://huggingface.co/spaces/mteb/leaderboard