Gathering usage statistics

[telepenin]

Hi,
I've enabled pr-agent (gerrit git provider) for bunch of repositories and I would like to gather some statistics.
Fields might be interesting for me: project, pull request id, action point (/review, /describe, etc.), prompts, result, tokens in, tokens out, costs.

Separately I ran my small project for review + generating improving where I'm using google sphreadsheets (via gspread) directly to store analytics info. Seems it's pretty nice to review how the project work for many multiples projects.

One more thing, pr-agent uses openai library directly, as I understand it does not provide counting tokens in, tokens out, costs for further analyse on the openai requests, but langchain does. It might be interesting as well.

[krrishdholakia]

Hey @telepenin,

the response object for all queries follows the openai object format which contains the tokens used for prompt and completion

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-3.5-turbo-0613",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "\n\nHello there, how may I assist you today?",
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
  }
}

Since pr-agent also uses litellm - you can also use that count cost per token and cost of the completion query.

I'm happy to make a PR to integrate these, if you can explain how you'd like to call / use these

[okotek]

/similar_issue

[github-actions]

Similar Issues:

1. Unable to use Github Action anymore

2. Running Github Action: openai.error.RateLimitError

3. [Feature Idea / Documentation] PR Agent interacts well with Github Saved Replies

[okotek]

/similar_issue

[github-actions]

Similar Issues:

1. Unable to use Github Action anymore

2. Running Github Action: openai.error.RateLimitError

3. [Feature Idea / Documentation] PR Agent interacts well with Github Saved Replies

[mrT23]

i am closing the issue

if still relevant, please re-open