-
Notifications
You must be signed in to change notification settings - Fork 555
Description
Problem Statement
Except tokens usage and latency metadata, here are some additional data may be returned by models API. For example, log probabilities of the output tokens. But Strands can't handle this type of metadata and display it to the user with the final response.
Proposed Solution
I'm not experienced in Strands "core" code, so can't provide direct implementation. But at the abstract layer, it's necessary to add something similar to aggregator for token usage. Because logprobs are returned in each chunk and app have to aggregate them into one huge logprobs data list.
Use Case
If the model can provide this type of metadata, in format_chunk and stream methods it will be possible to implement handling of this type of metadata and hand over it to Strands "core" code to aggregate it in the final response likewise tokens usage metadata, for example. For now, even if you try to return any additional metadata it won't be displayed in the final response
Alternatives Solutions
No response
Additional Context
No response