Skip to content

feat(google): add VertexRAGRetrieval provider tool#5222

Merged
tinalenguyen merged 1 commit intolivekit:mainfrom
youpesh:feat/retrieval-provider-tool
Mar 26, 2026
Merged

feat(google): add VertexRAGRetrieval provider tool#5222
tinalenguyen merged 1 commit intolivekit:mainfrom
youpesh:feat/retrieval-provider-tool

Conversation

@youpesh
Copy link
Copy Markdown
Contributor

@youpesh youpesh commented Mar 25, 2026

Summary

Adds a VertexRAGRetrieval provider tool that enables server-side retrieval via Vertex AI RAG Engine during Gemini inference. Retrieval happens in a single inference pass with no tool-call round-trip — critical for low-latency voice agents using native audio.

Usage

from livekit.plugins import google

agent = MyAgent(
    llm=google.realtime.RealtimeModel(
        model="gemini-live-2.5-flash-native-audio",
        vertexai=True,
    ),
    tools=[
        google.tools.VertexRAGRetrieval(
            rag_resources=["projects/123/locations/us-central1/ragCorpora/456"],
            similarity_top_k=3,
        ),
    ],
)

Motivation

Vertex AI RAG Engine works like Google Search grounding — retrieval is handled server-side during inference. With function-calling RAG, Gemini native audio requires 2 inference passes (~3-5s). With server-side retrieval, it's a single pass (~0.5s).

The Retrieval / VertexRagStore types already exist in google.genai.types and the Gemini Live API supports them, but there was no provider tool wrapper to use them through the LiveKit agents framework.

Changes

  • Added VertexRAGRetrieval dataclass to tools.py following the same GeminiTool pattern as GoogleSearch, FileSearch, etc.

Closes #5221

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Mar 25, 2026

CLA assistant check
All committers have signed the CLA.

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

Copy link
Copy Markdown
Member

@tinalenguyen tinalenguyen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you for the PR!!

@tinalenguyen tinalenguyen merged commit 0708c2c into livekit:main Mar 26, 2026
14 checks passed
osimhi213 added a commit to de-id/livekit-agents that referenced this pull request Mar 29, 2026
…endor

* upstream/main:
  Baseten Plugin Update: fix metadata schema, add chain_id support, and improve response parsing (livekit#4889)
  feat(anthropic): support strict tool use schema (livekit#5259)
  feat(mistral): add voxtral TTS support (livekit#5245)
  Fix/stt fallback adapter propagate aligned transcript (livekit#5237)
  fix: expose endpointing_opts in AgentSession.update_options() (livekit#5243)
  fix(utils): improve type annotation for deprecate_params decorator (livekit#5244)
  (google realtime): add gemini-3.1-flash-live-preview model (livekit#5233)
  fix: Nova Sonic interactive context bugs and dynamic tool support (livekit#5220)
  feat(assemblyai): add domain parameter for Medical Mode (livekit#5208)
  fix: ensure MCP client enter/exit run in the same task (livekit#5223)
  feat(google): add VertexRAGRetrieval provider tool (livekit#5222)
osimhi213 added a commit to de-id/livekit-agents that referenced this pull request Mar 29, 2026
* upstream/main:
  Baseten Plugin Update: fix metadata schema, add chain_id support, and improve response parsing (livekit#4889)
  feat(anthropic): support strict tool use schema (livekit#5259)
  feat(mistral): add voxtral TTS support (livekit#5245)
  Fix/stt fallback adapter propagate aligned transcript (livekit#5237)
  fix: expose endpointing_opts in AgentSession.update_options() (livekit#5243)
  fix(utils): improve type annotation for deprecate_params decorator (livekit#5244)
  (google realtime): add gemini-3.1-flash-live-preview model (livekit#5233)
  fix: Nova Sonic interactive context bugs and dynamic tool support (livekit#5220)
  feat(assemblyai): add domain parameter for Medical Mode (livekit#5208)
  fix: ensure MCP client enter/exit run in the same task (livekit#5223)
  feat(google): add VertexRAGRetrieval provider tool (livekit#5222)
osimhi213 added a commit to de-id/livekit-agents that referenced this pull request Mar 29, 2026
* feat(google): add VertexRAGRetrieval provider tool (livekit#5222)

* fix: ensure MCP client enter/exit run in the same task (livekit#5223)

* feat(assemblyai): add domain parameter for Medical Mode (livekit#5208)

* fix: Nova Sonic interactive context bugs and dynamic tool support (livekit#5220)

Co-authored-by: Pavas Kant <pavkan@amazon.com>

* (google realtime): add gemini-3.1-flash-live-preview model (livekit#5233)

* fix(utils): improve type annotation for deprecate_params decorator (livekit#5244)

* fix: expose endpointing_opts in AgentSession.update_options() (livekit#5243)

* Fix/stt fallback adapter propagate aligned transcript (livekit#5237)

* feat(mistral): add voxtral TTS support (livekit#5245)

* feat(anthropic): support strict tool use schema (livekit#5259)

* Baseten Plugin Update: fix metadata schema, add chain_id support, and improve response parsing (livekit#4889)

---------

Co-authored-by: Yousuf Bukhari <25112850+youpesh@users.noreply.github.com>
Co-authored-by: Long Chen <longch1024@gmail.com>
Co-authored-by: Martin Schweiger <34636718+m-ods@users.noreply.github.com>
Co-authored-by: Osman-AGI <uyguripek@gmail.com>
Co-authored-by: Pavas Kant <pavkan@amazon.com>
Co-authored-by: Tina Nguyen <72938484+tinalenguyen@users.noreply.github.com>
Co-authored-by: Milad <129620931+miladmnasr@users.noreply.github.com>
Co-authored-by: Jean Perbet <jeanperbet@icloud.com>
Co-authored-by: Shaik Faizan Roshan Ali <roshan.shaik.ml@gmail.com>
Co-authored-by: jiegong-fde <jie.gong@baseten.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Vertex AI RAG Engine (Retrieval) as a provider tool for Gemini Live

3 participants