Does LocalAI support models other than bert-MiniLM-L6-v2q4_0?

[zenyanbo]

Hello, Does LocalAI support models other than bert-MiniLM-L6-v2q4_0?
For example, bge-base-en-v1.5, nomic-embed-text-v1.5. Because the dimensions and maximum token of all-MiniLM-L6-v2 are too small, only 384 and 512 respectively. Might not be enough for RAG.

[mudler]

yes, it does support anything supported by sentencetransformers and transformers for embeddings - you need to switch the model name appropriately in the YAML config file. See also the docs here: https://localai.io/features/embeddings/#huggingface-embeddings

[zenyanbo]

Thank you. I'm always trying bert-embedding and llama.cpp embedding, sentencetransformers haven't tried yet.

[zenyanbo]

I found bge-m3 model which has 8k size and 1024 dimensions. I think it's useful.
Config:

name: text-embedding-3-large
backend: sentencetransformers
embeddings: true
parameters:
  model: BAAI/bge-m3

[ciekawy]

I managed to extend transformers backend and use bge-m3 as both embeddings (via openai route/schema) and reranker (via jina route/schema). The cool thing is it uses same model instance for both. Also I managed to easily enable "mps" for any transformer (this is worth separate PR)
This is now pretty hacky but maybe will manage to provide a PR.

[mudler]

That'd be great @ciekawy !