[Feature] DSPy Audio/Video Support Tracking

[isaacbmiller]

What feature would you like to see?

We have received a number of requests for Audio and Video input support over the last few months (#2037, #7844, etc.)

I implemented DSPy.Image, and am looking for someone to help out and create similar or better implementations for audio and/or video inputs. It would be shocking to me if some good prompting and few shot suppport for audio would greatly help in some use cases, and also being able to script with audio in the same way that you can with text inputs.

For someone to implement this, there are a few required steps for the implementation I am imagining:

1. create a class similar to Image (see adapters/image_utils.py)
2. edit chat_adapter and json_adapter to have a try_expand_audio_tags method that will search and expand messages with multimodal inputs
3. Write tests similar to tests/signatures/test_adapter_image.py to make sure it can work with a variety of signature types and input methods

I don't know much about the audio input APIs to really know what the speedbumps on this implementation are going to be.

As a first step, I would choose either the OpenAI API or Gemini, and get it working for that provider with whatever hacky code is needed, then expand and abstract after that.

feel free to @ me in the discord username is ibmiller if you need help

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[pretbc]

Yes, I'd like to help implement this.

[ramisbahi]

Yes, I'd like to help implement this: I'd like to help with implementing support for Video input. Additionally, I'm looking into benchmarking video understanding performance to evaluate how well DSPy can process video-based inputs

[glesperance]

This PR should work for audio (and more) #7872

After some review, the PR takes us closer to supporting audio but there's still work to be done to support this.
For reference here's the LiteLLM doc on using audio models[1]

[1] https://docs.litellm.ai/docs/completion/audio#audio-input-to-a-model

[pretbc]

Hello guys

for AZURE openAI we have to add in Image module

def expand_image_tags(text: str) -> Union[str, List[Dict[str, Any]]]:
    """Expand image tags in the text. If there are any image tags, 
    turn it from a content string into a content list of texts and image urls.
    
    Args:
        text: The text content that may contain image tags
        
    Returns:
        Either the original string if no image tags, or a list of content dicts
        with text and image_url entries
    """
    image_tag_regex = r'"?<DSPY_IMAGE_START>(.*?)<DSPY_IMAGE_END>"?'
    
    # If no image tags, return original text
    if not re.search(image_tag_regex, text):
        return text
        
    final_list = []
    remaining_text = text
    
    while remaining_text:
        match = re.search(image_tag_regex, remaining_text)
        if not match:
            if remaining_text.strip():
                final_list.append({"type": "text", "text": remaining_text.strip()})
            break
            
        # Get text before the image tag
        prefix = remaining_text[:match.start()].strip()
        if prefix:
            final_list.append({"type": "text", "text": prefix})
            
        # Add the image
        image_url = match.group(1)
        # final_list.append({"type": "image_url", "image_url": {"url": image_url}})

        data = image_url.split(",", 1)[1]  # Get the base64 encoded data
        final_list.append({
            "type": "input_audio",
            "input_audio": {
                "data": data,
                "format": 'wav'
            }
        })
        
        # Update remaining text
        remaining_text = remaining_text[match.end():].strip()
    
    return final_list

due to fact azure use for audio models

completion = client.chat.completions.create( 
    model="gpt-4o-mini-audio-preview", 
    modalities=["text", "audio"], 
    audio={"voice": "alloy", "format": "wav"}, 
    messages=[ 
        { 
            "role": "user", 
            "content": [ 
                {  
                    "type": "text", 
                    "text": "Describe in detail the spoken audio input." 
                }, 
                { 
                    "type": "input_audio", 
                    "input_audio": { 
                        "data": encoded_string, 
                        "format": "wav" 
                    } 
                } 
            ] 
        }, 
    ] 
) 

MIME like generate cause errors 'audio/x-wav'.

litellm.BadRequestError: AzureException BadRequestError - Invalid value: 'audio/x-wav'. Supported values are: 'wav' and 'mp3'.

I have no idea at which level we should check how to handle data. Best would be to check for provider type and create some sub pipeline to process final_list.append

[arnavsinghvi11]

Closed via #8214 #8231

[pushkarparanjpe]

Yes, I'd like to help implement this: I'd like to help with implementing support for Video input. Additionally, I'm looking into benchmarking video understanding performance to evaluate how well DSPy can process video-based inputs

Hey @ramisbahi - curious to know if video support was implemented.

• If so - Is this available in the main branch ?
• How did you do it - for instance did you rollout a dspy.Video class ?
• I am interested in using video input to Google VertexAI Gemini models. Any help in that direction is much appreciated.

Thanks!

[TirushV]

Yes, I'd like to help implement this: I'd like to help with implementing support for Video input. Additionally, I'm looking into benchmarking video understanding performance to evaluate how well DSPy can process video-based inputs

Hey @ramisbahi - curious to know if video support was implemented.

• If so - Is this available in the main branch ?
• How did you do it - for instance did you rollout a dspy.Video class ?
• I am interested in using video input to Google VertexAI Gemini models. Any help in that direction is much appreciated.

Thanks!

+1