[Platform] Standardise token usage

[junaidbinfarooq]

Q	A
Bug fix?	no
New feature?	yes
Docs?	yes
Issues	#208
License	MIT

Changes proposed:

The PR aims to standardize the token usage extraction and its availability in the AI bundle. Essentially, the following is done:

• Makes token usage explicit by adding a dedicated dto for token information
• Populates the token usage dto inside different token output processors and then adds it to the metadata object

[chr-hertel]

Sorry, but we need to slim this down a bit - in general smaller PRs, that build on top of each other are favorable to me. so let's break it down like you did:

Makes token usage explicit

I like having the value object, we should definitely have that 👍
But I'm not in favor of having the getTokenUsage() directly on the result since the result is a high lever abstraction and token usage is quite specific to remote inference of GenAI models - not all models have that. let's keep it in the metadata please.

Adds support to automatically register token usage extractors so that the token usage information is available in the result when a model is invoked

That's a bit misleading since token usage was already sitting on an extension point => OutputProcessor and your design decision to use that extension point to add another extension point doesn't really resonate with me. why is the new extension point needed? it feels like we're just adding them because we can.

Makes the token usage auto-registration configurable

For me this is too early, i don't think that a use-case, do you? how often do we see the case for someone to bring their own token usage extractor.

---

all in all, let's start here please with the TokenUsage value object in the metadata class - that's an easy step we see the same picture 👍

edit: another PR could be to fix that issue #208 by registering the corresponding output processor based on the config setting

[junaidbinfarooq]

@chr-hertel

Makes token usage explicit
I like having the value object, we should definitely have that 👍
But I'm not in favor of having the getTokenUsage() directly on the result since the result is a high lever abstraction and token usage is quite specific to remote inference of GenAI models - not all models have that. let's keep it in the metadata please.

Hmm, I initially thought about keeping it in metadata, but having it inside the Result object also made sense since an API should send us this information. I, so far, have seen such information in the responses of whatever APIs I tried, but perhaps you are right, and other models out there do not provide such information.

Adds support to automatically register token usage extractors so that the token usage information is available in the result when a model is invoked
That's a bit misleading since token usage was already sitting on an extension point => OutputProcessor and your design decision to use that extension point to add another extension point doesn't really resonate with me. why is the new extension point needed? it feels like we're just adding them because we can.

Nope, the idea, instead, was to introduce an extension point specific to token usage extraction and exposure. This would also simplify for bridges to their own implentations of the extractor interface if needed.

Makes the token usage auto-registration configurable
For me this is too early, i don't think that a use-case, do you? how often do we see the case for someone to bring their own token usage extractor.

So far, in this repository, I have found three such extractors (currently TokenOutputProcessor classes), including the new Google Vertex AI bridge, and all three of them perform the extraction differently, albeit not entirely. As you also rightly pointed out above, token usage is something specific to GenAI models, and not all of them have it. Not all of them will expose the data in a similar fashion, either.

[junaidbinfarooq]

@OskarStark @chr-hertel
I have reverted the previous changes and only kept the token usage information. Also, edited the PR description.
Please review the changes.

[OskarStark]

What about having a dedicated setTokenUsage/getTokenUsage on the metadata object?

[junaidbinfarooq]

Not sure about this.
Metadata object already has a lot of methods, and each of them is generic in nature as far as I can see, and doesn't concern any specific model characteristic.

[chr-hertel]

almost done i'd say - thanks already!

[junaidbinfarooq]

almost done i'd say - thanks already!

I am working on autoconfiguring token usage processors inside the AI bundle and based that branch on this one.
A PR will arrive after this one is finished.

[chr-hertel]

Good to merge from my end - @OskarStark?

[junaidbinfarooq]

Good to merge from my end - @OskarStark?

@OskarStark
Could you have a look?

[chr-hertel]

Thank you @junaidbinfarooq.