Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions docs/docs/ai/llm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ We support the following types of LLM APIs:
| [Anthropic](#anthropic) | `LlmApiType.ANTHROPIC` | ✅ | ❌ |
| [Voyage](#voyage) | `LlmApiType.VOYAGE` | ❌ | ✅ |
| [LiteLLM](#litellm) | `LlmApiType.LITE_LLM` | ✅ | ❌ |
| [OpenRouter](#openrouter) | `LlmApiType.OPEN_ROUTER` | ✅ | |
| [OpenRouter](#openrouter) | `LlmApiType.OPEN_ROUTER` | ✅ | |
| [vLLM](#vllm) | `LlmApiType.VLLM` | ✅ | ❌ |
| [Bedrock](#bedrock) | `LlmApiType.BEDROCK` | ✅ | ❌ |

Expand Down Expand Up @@ -400,7 +400,7 @@ You can find the full list of models supported by LiteLLM [here](https://docs.li
To use the OpenRouter API, you need to set the environment variable `OPENROUTER_API_KEY`.
You can generate the API key from [here](https://openrouter.ai/settings/keys).

A spec for OpenRouter looks like this:
A text generation spec for OpenRouter looks like this:

<Tabs>
<TabItem value="python" label="Python" default>
Expand All @@ -415,6 +415,27 @@ cocoindex.LlmSpec(
</TabItem>
</Tabs>

OpenRouter also supports some text embedding models. Note that for OpenRouter embedding
models, you need to explicitly provide the `output_dimension` parameter in the spec.
Here's how you can define the spec to use an OpenRouter embedding model:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.functions.EmbedText(
api_type=cocoindex.LlmApiType.OPEN_ROUTER,
model="openai/text-embedding-3-small",
# Task type for embedding model
task_type="SEMANTICS_SIMILARITY",
# Required: the number of output dimensions for the embedding model
output_dimension=1536,
)
```

</TabItem>
</Tabs>

You can find the full list of models supported by OpenRouter [here](https://openrouter.ai/models).

### vLLM
Expand Down
10 changes: 5 additions & 5 deletions rust/cocoindex/src/llm/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,10 @@ pub async fn new_llm_embedding_client(
LlmApiType::Ollama => {
Box::new(ollama::Client::new(address).await?) as Box<dyn LlmEmbeddingClient>
}
LlmApiType::OpenRouter => {
Box::new(openrouter::Client::new_openrouter(address, api_key).await?)
as Box<dyn LlmEmbeddingClient>
}
LlmApiType::Gemini => {
Box::new(gemini::AiStudioClient::new(address, api_key)?) as Box<dyn LlmEmbeddingClient>
}
Expand All @@ -178,11 +182,7 @@ pub async fn new_llm_embedding_client(
Box::new(gemini::VertexAiClient::new(address, api_key, api_config).await?)
as Box<dyn LlmEmbeddingClient>
}
LlmApiType::OpenRouter
| LlmApiType::LiteLlm
| LlmApiType::Vllm
| LlmApiType::Anthropic
| LlmApiType::Bedrock => {
LlmApiType::LiteLlm | LlmApiType::Vllm | LlmApiType::Anthropic | LlmApiType::Bedrock => {
api_bail!("Embedding is not supported for API type {:?}", api_type)
}
};
Expand Down
Loading