Best way to use an OpenAI-compatible embedding API

[BeautyyuYanli]

Hello everyone!

I'm using my own OpenAI-compatible embedding API, the runnable code:

from llama_index.embeddings.openai import OpenAIEmbedding

emb_model = OpenAIEmbedding(
    api_key="DUMMY_API_KEY",
    api_base="http://192.168.0.1:8000",
    model_name="intfloat/multilingual-e5-large",
)
emb = emb_model.get_text_embedding("hello world")
print(emb)

But if I use the arg model instead of model_name, It will raise:

ValueError: 'intfloat/multilingual-e5-large' is not a valid OpenAIEmbeddingModelType

I checked the code in llama_index/embeddings/openai/base.py,

class OpenAIEmbedding(BaseEmbedding):
    ...
    def __init__(
        ...
        self._query_engine = get_engine(mode, model, _QUERY_MODE_MODEL_DICT)
        self._text_engine = get_engine(mode, model, _TEXT_MODE_MODEL_DICT)

        if "model_name" in kwargs:
            model_name = kwargs.pop("model_name")
            self._query_engine = self._text_engine = model_name
        else:
            model_name = model

So I'm not sure if it is a good idea to set the model_name arg to use my custom OpenAI-compatible API? If not, is there a better way? If yes, maybe we need to add some documents?

[dosubot[bot]]

Hey there @BeautyyuYanli! Good to see you back in the world of embeddings 🌐🦙

Based on the code and context you've provided, it seems like you're on the right track. Using the model_name argument to specify your custom OpenAI-compatible API is a valid approach. This is because the model_name parameter allows for directly specifying the embedding engine without going through the mapping logic that uses the model parameter.

In the OpenAIEmbedding class, if model_name is provided in the keyword arguments (kwargs) during the initialization, it overrides the engine selection logic based on the model parameter. Both the _query_engine and _text_engine attributes are set directly to the value of model_name. This provides flexibility for users who know exactly which engine they want to use or need to use an engine that doesn't fit the predefined model types.

Here's the relevant part of the code from llama_index/embeddings/openai/base.py:

class OpenAIEmbedding(BaseEmbedding):
    ...
    def __init__(
        ...
        self._query_engine = get_engine(mode, model, _QUERY_MODE_MODEL_DICT)
        self._text_engine = get_engine(mode, model, _TEXT_MODE_MODEL_DICT)

        if "model_name" in kwargs:
            model_name = kwargs.pop("model_name")
            self._query_engine = self._text_engine = model_name
        else:
            model_name = model

As for the documentation, I agree with you. It would be beneficial to have more detailed documentation about the usage of model and model_name parameters in the OpenAIEmbedding class. This would help users to better understand how to use these parameters and how they affect the behavior of the class. I recommend opening an issue in the LlamaIndex repository to suggest this improvement.

I hope this helps! If you have any more questions, feel free to ask.

Sources

• llama-index-legacy/llama_index/legacy/llms/openai.py
• llama-index-legacy/llama_index/legacy/embeddings/openai.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[BeautyyuYanli]

Do you generate the answer by your illusion?

[Searcherr]

Hello everyone!

I'm using my own OpenAI-compatible embedding API, the runnable code:

from llama_index.embeddings.openai import OpenAIEmbedding

emb_model = OpenAIEmbedding(
    api_key="DUMMY_API_KEY",
    api_base="http://192.168.0.1:8000",
    model_name="intfloat/multilingual-e5-large",
)
emb = emb_model.get_text_embedding("hello world")
print(emb)

But if I use the arg model instead of model_name, It will raise:

ValueError: 'intfloat/multilingual-e5-large' is not a valid OpenAIEmbeddingModelType

I checked the code in llama_index/embeddings/openai/base.py,

class OpenAIEmbedding(BaseEmbedding):
    ...
    def __init__(
        ...
        self._query_engine = get_engine(mode, model, _QUERY_MODE_MODEL_DICT)
        self._text_engine = get_engine(mode, model, _TEXT_MODE_MODEL_DICT)

        if "model_name" in kwargs:
            model_name = kwargs.pop("model_name")
            self._query_engine = self._text_engine = model_name
        else:
            model_name = model

So I'm not sure if it is a good idea to set the model_name arg to use my custom OpenAI-compatible API? If not, is there a better way? If yes, maybe we need to add some documents?

Greetings!

Did you find the way to solve this issue?

Thank you in advance!

[juanluisrosaramos]

But this model is in hugginface no? if yes you need to load it with
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

[Searcherr]

But this model is in hugginface no? if yes you need to load it with from llama_index.embeddings.huggingface import HuggingFaceEmbedding
Thank you so much!

What if I created OpenAI-like web application with huggingface models?
So I want to call this API with, for example OpenAIEmbedding class (or similar). But change base_url to my endpoint and model_name to one of the huggingface models?

Thank you in advance!

[david1542]

Did anyone find a solution to this problem?

[Searcherr]

Some kind of yes. Using OllamaEmbedding object instead:
https://docs.llamaindex.ai/en/stable/examples/embeddings/ollama_embedding/

[vikrantdeshpande09876]

You might wanna check out TextEmbed.
It seems to support OpenAI-compatible Embedding API format.
However, I noticed the source code needed a base-URL change from /embedding to /embeddings.
Ended up recreating TextEmbed as my own custom Embedding class.

[WinPooh32]

I have found solution for OpenAI-compatible APIs. In my case it's LiteLLM as frontend and ollama as a backend api. Solution is just using langchain openai clients:

from llama_index.embeddings.langchain import LangchainEmbedding
from llama_index.llms.langchain import LangChainLLM
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

embed_model = LangchainEmbedding(
    langchain_embeddings=OpenAIEmbeddings(
        model="snowflake-arctic-embed2:latest",
        dimensions=1024,
        base_url="http://127.0.0.1:4000",
        api_key="key",
        timeout=500,
    ),
)

llm = LangChainLLM(
    llm=ChatOpenAI(
        model="qwen2.5-coder:32b-instruct-q4",
        base_url="http://127.0.0.1:4000",
        api_key="key",
        disable_streaming="tool_calling",
    )
)

Libs to install:

pip install llama-index-embeddings-langchain
pip install llama-index-llms-langchain
pip install langchain_openai
pip install langchain
pip install langchain_community

[logan-markewich]

@BeautyyuYanli @WinPooh32 @david1542 @juanluisrosaramos @vikrantdeshpande09876

Here's the non-langchain method for using openai-like API's

pip install llama-index-llms-openai-like llama-index-embeddings-openai

from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai_like import OpenAILike

embed_model = OpenAIEmbedding(
  model="some model", # use `model` instead of `model_name` -- janky I know
  api_base="...",
  api_key="fake",
  embed_batch_size=10,
)

llm = OpenAILike(
  model_name="my model",
  api_key="fake",
  api_base="...",
  # context window should match whatever llm you are using
  context_window=32000,  
  # specifies whether or not to use chat completions endpoint
  is_chat_model=True,
  # specifies if function calling over the API is supported
  is_function_calling_model=False,
)

[WinPooh32]

Ollama doesn't support streaming when tools are called. That's why LiteLLM fails to get response from Ollama, when I use OpenAILike. Also there are no options to disable streaming at this client. In my case langchain clients work perfectly.

Error from litellm:

litellm-1  | 10:00:06 - LiteLLM:ERROR: ollama_chat.py:508 - LiteLLM.ollama(): Exception occured - sequence item 18: expected str instance, NoneType found
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/lib/python3.13/site-packages/litellm/llms/ollama_chat.py", line 482, in ollama_async_streaming
litellm-1  |     response_content = first_chunk_content + "".join(content_chunks)

My confing:

llm = OpenAILike(
    model="qwen2.5-coder:32b-instruct-q4",
    api_key="somekey",
    api_base="http://127.0.0.1:4000/v1",
    # context window should match whatever llm you are using
    context_window=8191,
    # specifies whether or not to use chat completions endpoint
    is_chat_model=True,
    # specifies if function calling over the API is supported
    is_function_calling_model=True,
    timeout=500,
)

[WinPooh32]

I have connected Ollama to LiteLLM using OpenAI-api and problem with OpenAILike is gone.
BerriAI/litellm#7094 (comment)

[WinPooh32]

But it's impossible to set non-openai model when OpenAIEmbedding client is used due to enum:

    key = (OpenAIEmbeddingMode(mode), OpenAIEmbeddingModelType(model))
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/enum.py", line 757, in __call__
    return cls.__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/enum.py", line 1171, in __new__
    raise ve_exc
ValueError: 'nomic-embed-text:v1.5' is not a valid OpenAIEmbeddingModelType

My code:

from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    model="nomic-embed-text:v1.5",
    dimensions=768, 
    embedding_ctx_length=8191,
    api_base="http://127.0.0.1:4000",
    api_key="somekey",
)

[logan-markewich]

@BeautyyuYanli sounds like an issue for litellm then? Our official Ollama llm class works fine with streaming tool calls 😁

And my bad for the openai embeddings, you should use model_name= not model=