Skip to content

feat(embedding): add litellm as embedding provider#853

Merged
ZaynJarvis merged 2 commits intovolcengine:mainfrom
stubbi:feat/litellm-embedding-provider
Mar 22, 2026
Merged

feat(embedding): add litellm as embedding provider#853
ZaynJarvis merged 2 commits intovolcengine:mainfrom
stubbi:feat/litellm-embedding-provider

Conversation

@stubbi
Copy link
Contributor

@stubbi stubbi commented Mar 21, 2026

Summary

Adds LiteLLM as a new embedding provider, resolving the gap between VLM (which already supports litellm) and the embedding layer. This enables users to route embedding requests through OpenRouter, Ollama, vLLM, and any OpenAI-compatible endpoint via litellm's unified interface.

  • New file: openviking/models/embedder/litellm_embedders.pyLiteLLMDenseEmbedder class extending DenseEmbedderBase
  • Updated: embedding_config.py — added "litellm" to the provider validation list, factory registry, and docstrings
  • Updated: embedder/__init__.py — added conditional import (graceful fallback if litellm not installed)
  • New tests: tests/unit/test_litellm_embedder.py — 15 tests covering embed, batch embed, non-symmetric mode, factory integration, and config validation

Usage example

"embedding": {
  "dense": {
    "provider": "litellm",
    "model": "openai/text-embedding-3-small",
    "api_base": "https://openrouter.ai/api/v1",
    "api_key": "<openrouter-key>",
    "dimension": 1536
  }
}

Closes #847

Test plan

  • All 15 new unit tests pass
  • All existing embedding tests still pass (no regressions)
  • Manual test with OpenRouter endpoint
  • Manual test with local Ollama embeddings

🤖 Generated with Claude Code

Adds LiteLLM as a new embedding provider, bringing embedding parity with
the VLM layer which already supports litellm. This enables users to route
embedding requests through OpenRouter, Ollama, vLLM, and any other
OpenAI-compatible endpoint via litellm's unified interface.

Closes volcengine#847

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions
Copy link

Failed to generate code suggestions for PR

Copy link
Collaborator

@ZaynJarvis ZaynJarvis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical Issues Found

1. Module-level side effect on import (litellm_embedders.py:13)

os.environ.setdefault("LITELLM_LOCAL_MODEL_COST_MAP", "True")

This mutates the process environment at import time — not when the embedder is used. Since __init__.py imports this module (behind try/except), this fires as soon as the embedder package is loaded, even if the user never uses litellm. This can interfere with other litellm usage in the same process that intentionally sets this env var differently.

Fix: Move this inside LiteLLMDenseEmbedder.__init__ or into _build_kwargs.

2. Probe API call during __init__ / silent fallback (litellm_embedders.py:89-93)

def _detect_dimension(self) -> int:
    try:
        result = self.embed("test")
        return len(result.dense_vector) if result.dense_vector else 1536
    except Exception:
        return 1536

When dimension=None, the constructor calls _detect_dimension() which calls self.embed("test"). This means:

  1. A billable API call the user didn't ask for, fired during object construction
  2. A network side effect inside __init__ — surprising and hard to test
  3. Silent fallback to 1536 on any exception (auth failure, network error, wrong model) — the user gets no indication their config is wrong

Fix: Require dimension as a mandatory parameter (other providers do this), or at minimum make the probe call lazy and log a warning on fallback instead of silently swallowing errors.

3. Missing None guard for LiteLLMDenseEmbedder in factory

The factory in embedding_config.py imports LiteLLMDenseEmbedder but doesn't check if it's None (which happens when litellm isn't installed). Other providers like Gemini have this guard. If a user configures provider: "litellm" without litellm installed, they'll get a confusing TypeError: 'NoneType' object is not callable instead of a clear error message.

1. Move os.environ.setdefault from module-level to __init__ to avoid
   mutating the process environment on import
2. Require dimension as mandatory — removes the probe API call during
   construction that caused surprise billable requests and silent fallbacks
3. Add None guard for LiteLLMDenseEmbedder in factory to give a clear
   error when litellm is not installed

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@stubbi
Copy link
Contributor Author

stubbi commented Mar 22, 2026

Thanks for the review @ZaynJarvis! All three issues have been addressed in the latest push:

  1. Module-level side effectos.environ.setdefault("LITELLM_LOCAL_MODEL_COST_MAP", "True") moved from module scope into LiteLLMDenseEmbedder.__init__, so it only fires when an embedder is actually instantiated.

  2. Probe API call / silent fallback — Removed _detect_dimension() entirely. dimension is now required at both the config validation layer (EmbeddingModelConfig) and the embedder constructor, with clear error messages if missing.

  3. Missing None guard — The factory now checks if LiteLLMDenseEmbedder is None before attempting to instantiate, raising ValueError("LiteLLM is not installed. Install it with: pip install litellm") instead of a confusing TypeError.

Added 3 new tests covering the new validation paths (18 total, all passing).

Copy link
Collaborator

@ZaynJarvis ZaynJarvis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ZaynJarvis ZaynJarvis merged commit 49a50fd into volcengine:main Mar 22, 2026
4 of 6 checks passed
@github-project-automation github-project-automation bot moved this from Backlog to Done in OpenViking project Mar 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Support litellm (or OpenRouter) as embedding provider

3 participants