Skip to content

[FEATURE] Add support of logprobs metadata support #346

@yanomaly

Description

@yanomaly

Problem Statement

Except tokens usage and latency metadata, here are some additional data may be returned by models API. For example, log probabilities of the output tokens. But Strands can't handle this type of metadata and display it to the user with the final response.

Proposed Solution

I'm not experienced in Strands "core" code, so can't provide direct implementation. But at the abstract layer, it's necessary to add something similar to aggregator for token usage. Because logprobs are returned in each chunk and app have to aggregate them into one huge logprobs data list.

Use Case

If the model can provide this type of metadata, in format_chunk and stream methods it will be possible to implement handling of this type of metadata and hand over it to Strands "core" code to aggregate it in the final response likewise tokens usage metadata, for example. For now, even if you try to return any additional metadata it won't be displayed in the final response

Alternatives Solutions

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-providerRelated to model providersenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions