Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Hope to get the token usage report per AOAI connection deployment #2353

Open
Wix97 opened this issue Mar 14, 2024 · 4 comments
Assignees

Comments

@Wix97
Copy link

Wix97 commented Mar 14, 2024

For now the token usage report is summarized per batch run.
However, it would be more helpful for users to estimate costs if promptflow could provide token usage reports for each AOAI connection deployment.

It would also be helpful if the user can read such report in json format via PF SDK.

@Wix97 Wix97 added the enhancement New feature or request label Mar 14, 2024
@brynn-code
Copy link
Contributor

Hi @Wix97 . Thanks for reaching us, could you please elaborate more about the 'usage of each AOAI connection deployment'? Is that related to any run, for example a batch run with different deployment, to get the usage of each deployment, or just overall token usage of a deployment? For the latter one, seems that should be the capability of AzureOpenAI instead of promptflow.

@Wix97
Copy link
Author

Wix97 commented Mar 15, 2024

Sorry for the confusion, the first case is what I wanted to mention here.
For example, there is a flow consuming 2 AOAI connections, each connection has 2 deployments. It would be helpful if promptflow can generate token usage report such like:

  "token_usage": {
    "connection_1": {
      "connection_1_deployment_1": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      },
      "connection_1_deployment_2": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      }
    },
    "connection_2": {
      "connection_2_deployment_1": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      },
      "connection_2_deployment_2": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      }
    }
  }
}

@thy09
Copy link
Contributor

thy09 commented Mar 15, 2024

@Wix97 From pf perspective, it would be complicated to define such a contract to summarize tokens according to the tokens.

I have a proposal: what if we expose the ability to get the tokens in other nodes?
For example, we provide such an experession: {llm_node_1.system_metrics.usage} to get the usage dictionary of a node, then you can have a summary node to get the dictionary for yourself.

@Wix97
Copy link
Author

Wix97 commented Mar 17, 2024

LGTM👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants