Gemini API not supported

[dataopsnick]

Objective: Extend the existing Lambda proxy to support Google Gemini API alongside the existing OpenAI functionality, using the provided reference implementation as the gold standard.

Constraints:

• Do not remove or break existing OpenAI functionality.
• Use @google/generative-ai library for Gemini interaction.
• The URL endpoint is NOT needed, as the client lib handles that.
• GOLD STANDARD: Use the following code as the exact way to interact with the Gemini API. Mimic this code precisely when implementing the Gemini functionality.
• DO NOT MODIFY OpenAiSettings{}. It is SPECIFICALLY named "OpenAiSettings" for a reason, and should only be used for settings related to OpenAI.

import { GoogleGenerativeAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: "GEMINI_API_KEY" });

async function main() {
  const chat = ai.chats.create({
    model: "gemini-2.0-flash",
    history: [
      {
        role: "user",
        parts: [{ text: "Hello" }],
      },
      {
        role: "model",
        parts: [{ text: "Great to meet you. What would you like to know?" }],
      },
    ],
  });

  const stream1 = await chat.sendMessageStream({
    message: "I have 2 dogs in my house.",
  });
  for await (const chunk of stream1) {
    console.log(chunk.text);
    console.log("_".repeat(80));
  }

  const stream2 = await chat.sendMessageStream({
    message: "How many paws are in my house?",
  });
  for await (const chunk of stream2) {
    console.log(chunk.text);
    console.log("_".repeat(80));
  }
}

await main();

Steps (with STRONG reference to the example code):

1. Install @google/generative-ai: npm install @google/generative-ai

2. Update openai_servers.yaml: Add a gemini entry. Key elements to mimic from the example:

   • token: Your Gemini API key
   • model: The Gemini model name (e.g., "gemini-1.5-pro-latest" or "gemini-2.0-flash" from the example). The example explicitly uses a model parameter in ai.chats.create, so include it.
   • Omit the url.
   • DO NOT INCLUDE provider: gemini. The name "gemini" in the yaml is the implicit provider.

   gemini:
     token: YOUR_GEMINI_API_KEY
     model: gemini-2.0-flash # Use the example's model for consistency

3. Create src/gemini_settings.ts:

   • Create a new interface named GeminiSettings with token and model properties. The keys should match the openai_servers.yaml.

   export interface GeminiSettings {
       token: string;
       model: string;
   }

4. Modify src/llm_client.ts:

   • Import OpenAi from openai and GoogleGenerativeAI from @google/generative-ai.
   • Import GeminiSettings from ./gemini_settings.
   • Modify constructor:
     • Take in settings: OpenAiSettings | GeminiSettings as a parameter.
     • Have two class properties, openaiSettings: OpenAiSettings and geminiSettings: GeminiSettings | null
     • If settings includes a url property then the object is of type OpenAiSettings, otherwise it's a GeminiSettings object.
     • Initialize openai client ONLY if settings is OpenAiSettings.
     • Initialize gemini client ONLY if settings is GeminiSettings.
     • Store the model: this.model = settings.model; (as done in the reference).
   • chatCompletionStreaming(content: string): This is the critical part.
     • Create a GenerativeModel instance: const model = this.gemini.getGenerativeModel({ model: this.model }); (exactly as in the example, passing the model from the config).
     • Create a ChatSession instance: const chat = model.startChat({ history: [] });. Note: The example uses history: []. For now, keep it simple and do the same. IMPORTANT: Future Enhancement: This is where you would need to integrate the previous messages in the conversation, if you want to maintain a conversation history.
     • Call sendMessageStream: const result = await chat.sendMessageStream({ message: content });. Note: Precisely mimic the example's use of { message: content }. This is key. content is the user's message (the prompt).
     • Return the stream: return result.stream;
   • createCompletionStreaming(params: any): Remove this. The streaming logic is now within chatCompletionStreaming which is correct according to the example.
   • chatCompletionNonStreaming(content: string): Implement if needed, but focus on streaming first.
   • Other methods: Remove/adjust unused functions that are no longer valid.

5. Modify src/llm_proxy.ts:

   • Adapt transformGenerator function: This will depend on the exact structure of the stream coming back from gemini.sendMessageStream. You will need to carefully examine the chunk objects to extract the text.

   export const transformGenerator = async function*<F, T>(iterator: AsyncIterator<F>, transform: (f: F) => T) {
     while (true) {
       const next = await iterator.next();
       if (next.done) { return; }
       // Check what the shape of 'next.value' is, and adapt the transform function accordingly
       yield transform(next.value);
     }
   }

   • Also needs to import GeminiSettings
   • Also needs to change signature of getLlmClient method to getLlmClient(server: string): LlmClient {

6. Modify src/app_settings.ts:

   • Also needs to check if url key is included to determine if the setting should be of type OpenAiSettings or GeminiSettings.
   • Also needs to import GeminiSettings

7. Update Test file (src/tests/index.test.ts):

   • Add integration and unit tests for gemini server. Test streaming thoroughly.

8. Build and Deploy: npm run build followed by sam deploy --guided

Revised src/llm_client.ts (Illustrative - Focus on Streaming):

import OpenAi from 'openai';
import { OpenAiSettings } from './openai_settings';
import { GeminiSettings } from './gemini_settings';
import { GoogleGenerativeAI } from "@google/genai";

export class LlmClient {
    private openai: OpenAi | null;
    private gemini: GoogleGenerativeAI | null;
    private model: string;
    private openaiSettings: OpenAiSettings | null;
    private geminiSettings: GeminiSettings | null;


    constructor(settings: OpenAiSettings | GeminiSettings) {
        if ('url' in settings) {
            // It's OpenAiSettings
            this.openaiSettings = settings;
            this.geminiSettings = null;
            this.openai = new OpenAi({
                baseURL: settings.url,
                apiKey: settings.token,
            });
            this.gemini = null;
        } else {
            // It's GeminiSettings
            this.geminiSettings = settings;
            this.openaiSettings = null;
            this.gemini = new GoogleGenerativeAI(settings.token);
            this.openai = null;
        }

        this.model = settings.model;
    }

    async chatCompletionStreaming(content: string): Promise<any> {
        if (this.openai) {
            const params = {
                model: this.model,
                messages: [{ role: 'user', content: content }],
                stream: true
            };
            return await this.openai.chat.completions.create(params);
        } else if (this.gemini) {
            const model = this.gemini.getGenerativeModel({ model: this.model });
            const chat = model.startChat({
                history: [], // IMPORTANT: Add history here later
            });

            const result = await chat.sendMessageStream({ message: content }); // Mimic example

            return result.stream;
        } else {
            throw new Error(`Unsupported provider: ${this.provider}`);
        }
    }
}