Langfuse support for Audio prompt input / output

[g00dsec]

Describe your question

Is there Langfuse support for speech-to-text / text-to-speech models? Is there documentation to demonstrate speech-to-text / text-to-speech implementation?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.148.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Yes, Langfuse supports audio input and output for speech-to-text and text-to-speech models(1).

Audio Input and Output Support

Langfuse provides multi-modal tracing capabilities that include audio attachments(1). The platform can handle audio data through the LangfuseMedia object for efficient storage and retrieval(1).

Documentation and Examples

The multi-modality documentation demonstrates audio implementation with the OpenAI SDK(1). Here's an example showing audio input and output:

from langfuse import get_client

content_path = "static/joke_prompt.wav"

base64_string = encode_file(content_path)

response = client.chat.completions.create(
    model="gpt-4o-audio-preview",
    modalities=["text", "audio"],
    audio={"voice": "alloy", "format": "wav"},
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Do what this recording says."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": base64_string, "format": "wav"},
                },
            ],
        },
    ],
)

langfuse = get_client()
langfuse.flush()

(1)

Cost Tracking for TTS Models

For text-to-speech cost tracking, you need to log TTS runs as generation observations rather than span observations(2). Cost tracking in Langfuse only works for generation and embedding observation types(2).

To enable cost tracking for TTS:

1. Log TTS as a generation using @observe(as_type="generation") decorator or start_as_current_observation(as_type="generation") context manager(2)
2. Attach usage details that match your custom model definition (e.g., audio_duration, characters_count)(2)
3. Define a custom model in the Langfuse UI with usage types and prices matching the keys you send(2)

The usage detail keys in your code must exactly match the usage type keys in your Langfuse model definition for cost calculation to work(2).

📚 Sources:

1. Example: Multi-modality and attachments
2. Logging text-to-speech with Langfuse

---

Have another question? Just tag @inkeep.

[g00dsec]

@inkeep Can I know how Langfuse UI supports end-user prompt engineering for speech-to-text / text-to-speech modalities?

[jannikmaierhoefer]

@godclause, no, the Langfuse Playground and Langfuse in-UI Experiments do not support audio yet. We have this on our roadmap!

[inkeep-ai-assistant[bot]]

Based on the available sources, I don't have specific information about how the Langfuse UI supports end-user prompt engineering for speech-to-text or text-to-speech modalities.

The sources show that Langfuse supports multi-modal tracing with audio attachments(1), and you can log audio input/output in traces programmatically(1). However, the documentation doesn't describe UI features specifically designed for prompt engineering with audio modalities.

For general prompt management, there is a discussion about enhancing the Langfuse UI to support tool calls and agent configurations(2), but this doesn't address audio-specific prompt engineering capabilities.

If you need to work with audio prompts, you would currently need to:

• Use the programmatic SDK approach to log audio traces(1)
• Define custom models for cost tracking with audio-based usage metrics like audio_duration or characters_count(3)

For more detailed information about UI support for audio prompt engineering, I recommend checking the latest Langfuse documentation or reaching out to the Langfuse team directly, as this specific capability may not be fully documented in the available sources.

**

📚 Sources:

1. Example: Multi-modality and attachments
2. Support for Agents with Tool Calls in Langfuse Prompt Management UI
3. Logging text-to-speech with Langfuse

---

Have another question? Just tag @inkeep.