Skip to content

Include "id" and "model" in ChatResponseMetadata for better traceability #936

@ThomasVitale

Description

@ThomasVitale

Expected Behavior

Many LLM providers (including OpenAI, Mistral AI, and Anthropic), include an ID in their responses to uniquely identify an operation, and also include the exact model used to process the request (which differs from the one in the request, e.g. requested: gpt-3.5-turbo => actual: gpt-3.5-turbo-0125). I would expect them to be available in the ChatResponseMetadata interface. They will be critical when introducing observability in Spring AI (I'll soon share a draft solution about that, but this feature request is a pre-requisite).

Current Behavior

These two pieces of information are available in the lower LLM provider API, but they are not surfaced through the Spring AI abstractions. The "id" is actually already included in some customisations of ChatResponseMetadata, but it should be moved up to the generic interface.

Context

When using cloud-based LLM providers, it's important to be able to trace a specific operation for troubleshooting, evaluation, or further analysis. Knowing the unique operation ID and the actual model used to process the request helps towards that goal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions