Missing Instruction option in OpenAI Realtime

[khantseithu]

In previous 0.x version, we're allowed to configured instructions for openai realtime model.

model = openai.realtime.RealtimeModel(
                api_key=openai_api_key,
                model=config.model_name,
                voice=config.voice,
                temperature=config.temperature,
                # instructions=config.instructions, (this is missing)
                input_audio_transcription=InputAudioTranscription(
                    language='en',
                    model='gpt-4o-mini-transcribe',
                    prompt=combined_instructions # voice vibe does not work with this
                )
            )

My main problem is we've a feature called voice vibe using https://www.openai.fm/ which works by passing along to the system prompt. The instructions in AgentSession, Agent class doesn't work for voice vibe either. I noticed google.realtime.RealtimeModel has instruction field to be configurable, can we bring back instruction field to openai realtime as well.

[longcw]

Do you have an example "The instructions in AgentSession, Agent class doesn't work for voice vibe either."?

The instructions are supposed to be set from the agent session or agent. The google.realtime.RealtimeModel instruction is not intended, it will be overwritten by the instructions in agent session or agent.

[khantseithu]

@longcw Sure, here's an example with Agent class. I don't think AgentSession accepts instruction, sry about the mention previously. To reproduce, try it out with any prompts from https://www.openai.fm/. v1 version voice doesn't change according to the vibe.

Agent

class DemoAgent(Agent):
    def __init__(self, chatbot_id: str | None = None, instructions: str | None = None, stt: stt.STT | None = None,llm: llm.LLM | None = None, tts: tts.TTS | None = None, turn_detection: str | None = None, use_realtime_llm: bool = False):
        super().__init__(
            instructions=instructions or "You are a helpful voice AI assistant.",
            stt=stt,
            llm=llm,
            tts=tts,
            turn_detection=turn_detection,
            # vad=vad,
            # turn_detection=turn_detection
        )
        self.chatbot_id = chatbot_id
        self._session: aiohttp.ClientSession | None = None
        self._http_session: aiohttp.ClientSession | None = None  # For API calls like Cohere
        self.dynamic_tools: Dict[str, DynamicApiTool] = {}
        self.use_realtime_llm = use_realtime_llm
        logger.info(f"OkChatAgent initialized with chatbot_id: {chatbot_id}")
        
# other stuffs

main.py

        self.okchat_agent_instance = DemoAgent(
                chatbot_id=self.current_config.chatbot_id,
                instructions=combined_instructions,
                stt=self.stt,
                llm=self.llm,
                tts=self.tts,
                turn_detection=self.turn_detector,
                use_realtime_llm=use_realtime_llm,
            )
            
            # Initialize dynamic tools from the database
            await self.okchat_agent_instance.initialize()
            
            await self.current_agent.start(
                room=room,
                room_input_options=RoomInputOptions(
                    noise_cancellation=noise_cancellation.BVC(),
                ),
                agent=self.okchat_agent_instance,
                room_output_options=RoomOutputOptions(
                   transcription_enabled=True,
                   sync_transcription=False 
                )
            )

Previously in 0.x versions:

def create_model(self, config: SessionConfig) -> google.realtime.RealtimeModel | openai.realtime.RealtimeModel:
        if config.provider == "google":
            model = google.realtime.RealtimeModel(
                instructions=config.instructions,
                modalities=[Modality.AUDIO],
                voice=config.voice,
                temperature=config.temperature,
                # max_output_tokens=int(config.max_response_output_tokens),
                api_key=gemini_api_key,
                enable_user_audio_transcription=True,
                enable_agent_audio_transcription=True,
            )
        else:
            # For OpenAI model, check if vibe is provided and append it to instructions
            voice_instructions = (
                "Voice Instructions:" +
                "\n\n" +
                "Voice Affect: Calm, composed, and reassuring. Competent and in control, instilling trust. \n\n"
                "Tone: Sincere, empathetic, with genuine concern for the customer and understanding of the situation. \n\n"
                "Pacing: Slower during the apology to allow for clarity and processing. Faster when offering solutions to signal action and resolution. \n\n"
                "Emotions: Calm reassurance, empathy, and gratitude. \n\n"
                "Pronunciation: Clear, precise: Ensures clarity, especially with key details. Focus on key words like 'refund' and 'patience.' \n\n"
                "Pauses: Before and after the apology to give space for processing the apology."
            )
            
            # If vibe is provided, use it instead of the default voice instructions
            if config.vibe:
                logger.info('vibe %s', config.vibe)
                voice_instructions = f"Voice Instructions:\n\n{config.vibe}\n\n THIS IS THE VOICE VIBE, NOT THE CONTEXT OF THE CONVERSATION."
                
            model = openai.realtime.RealtimeModel(
                api_key=openai_api_key,
                model="gpt-4o-mini-realtime-preview",
                instructions=(
                    config.instructions + 
                    "\n\n" +
                    voice_instructions
                ),
                voice=config.voice,
                temperature=config.temperature,
                max_response_output_tokens=config.max_response_output_tokens,
                turn_detection=config.turn_detection,
            )
        return model

    def create_agent(self, model: google.realtime.RealtimeModel | openai.realtime.RealtimeModel, chat_ctx: llm.ChatContext, ctx: JobContext) -> MultimodalAgent:
        fnc_ctx = AssistantFnc(chatbot_id=self.current_config.chatbot_id, ctx=ctx)
        agent = MultimodalAgent(model=model, chat_ctx=chat_ctx, fnc_ctx=fnc_ctx)
        return agent

[longcw]

I didn't quite follow, you are using the pipeline agent with llm, stt, and tts, you can set a instructions for openai.TTS

tts=openai.TTS(instructions="", voice="...")

If you are using the realtime model (like your example for 0.x), you can set the instructions for the realtime model as well.

[davidzhao]

You are talking about TTS instructions. we support the exact same set of things for OpenAI's TTS: openai.TTS(instructions=...)

for a demo of it in action, take a look at https://www.livekit.fm/