refactor(*): change ai analytics #12583

AntoineJac · 2024-02-19T11:56:51Z

Summary

This will allow to add a provide layer in the analytics in case several plugins are used

Current model is:

  "ai": {
    "meta": {
      "response_model": "mistral-tiny",
      "provider_name": "mistral",
      "request_model": "mistral-tiny"
    },
    "usage": {
      "prompt_tokens": 259,
      "completion_tokens": 274,
      "total_tokens": 533
    }
  }

New model if using several plugins:

  "ai": {
    "mistral": {
      "request_total_tokens": 648,
      "number_of_instances": 1,
      "instances": [
        {
          "meta": {
            "request_model": "mistral-tiny",
            "plugin_id": "1c783122-0223-4a03-9f1d-c71d2baab01e",
            "response_model": "mistral-tiny",
            "provider_name": "mistral"
          },
          "usage": {
            "prompt_token": 392,
            "total_tokens": 648,
            "completion_token": 256
          }
        }
      ],
      "request_prompt_tokens": 392,
      "request_completion_tokens": 256
    },
    "azure": {
      "request_total_tokens": 145,
      "number_of_instances": 1,
      "instances": [
        {
          "meta": {
            "request_model": "gpt-35-turbo",
            "plugin_id": "5df193be-47a3-4f1b-8c37-37e31af0568b",
            "response_model": "gpt-35-turbo",
            "provider_name": "azure"
          },
          "usage": {
            "prompt_token": 89,
            "total_tokens": 145,
            "completion_token": 56
          }
        }
      ],
      "request_prompt_tokens": 89,
      "request_completion_tokens": 56
    },
    "cohere": {
      "request_total_tokens": 149,
      "number_of_instances": 1,
      "instances": [
        {
          "meta": {
            "request_model": "command",
            "plugin_id": "30a63ab9-03c6-447a-9b25-195eb0acaeb2",
            "response_model": "command",
            "provider_name": "cohere"
          },
          "usage": {
            "prompt_token": 78,
            "total_tokens": 149,
            "completion_token": 71
          }
        }
      ],
      "request_prompt_tokens": 78,
      "request_completion_tokens": 71
    }
  }

Checklist

The Pull Request has tests
A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #[issue number]

KAG-3759
KAG-4124

EDIT: format JSON snippets

CLAassistant · 2024-02-19T11:56:57Z

All committers have signed the CLA.

CLAassistant · 2024-02-19T11:56:57Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

AntoineJac · 2024-03-01T15:19:50Z

By using the AI request and response transformers you will get following logs:

When using same AI provider:

"ai": {
    "cohere": {
      "instances": [
        {
          "meta": {
            "response_model": "command",
            "provider_name": "cohere",
            "request_model": "command"
          },
          "usage": {
            "completion_tokens": 33,
            "prompt_tokens": 129,
            "total_tokens": 162
          }
        },
        {
          "meta": {
            "response_model": "command",
            "provider_name": "cohere",
            "request_model": "command"
          },
          "usage": {
            "completion_tokens": 25,
            "prompt_tokens": 412,
            "total_tokens": 437
          }
        }
      ],
      "request_prompt_tokens": 541,
      "request_completion_tokens": 58,
      "request_total_tokens": 599,
      "number_of_instances": 2
    }
  }

When using different AI providers:

  "ai": {
    "cohere": {
      "instances": [
        {
          "meta": {
            "response_model": "command",
            "provider_name": "cohere",
            "request_model": "command"
          },
          "usage": {
            "completion_tokens": 130,
            "prompt_tokens": 129,
            "total_tokens": 259
          }
        }
      ],
      "request_prompt_tokens": 129,
      "request_completion_tokens": 130,
      "request_total_tokens": 259,
      "number_of_instances": 1
    },
    "mistral": {
      "instances": [
        {
          "meta": {
            "response_model": "mistral-tiny",
            "provider_name": "mistral",
            "request_model": "mistral-tiny"
          },
          "usage": {
            "completion_tokens": 171,
            "prompt_tokens": 562,
            "total_tokens": 733
          }
        }
      ],
      "request_prompt_tokens": 562,
      "request_completion_tokens": 171,
      "request_total_tokens": 733,
      "number_of_instances": 1
    }
  }

EDIT: JSON formatting

subnetmarco · 2024-03-28T15:14:00Z

Would it be possible to also store the plugin_id for each instance so that the user can point back to where it's being used?

kong/llm/drivers/shared.lua

kong/plugins/ai-proxy/handler.lua

spec/03-plugins/38-ai-proxy/02-openai_integration_spec.lua

tysoekong

I've rebase'd and added a test - this is good to go now!

Co-authored-by: Joshua Schmid <jaiks@posteo.de>

AntoineJac · 2024-04-15T14:26:21Z

It seems we have resolved all the points. Are we ok to push this PR? @jschmid1
Thanks

team-gateway-bot · 2024-04-15T15:11:11Z

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-12583-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-12583-to-master-to-upstream
git checkout -b cherry-pick-12583-to-master-to-upstream
ancref=$(git merge-base 1b920ca144aed6669b2b1dbe5003683b4fcd18d3 9f3ab14f104e81d317aa6f2cc2d8337f5e72178a)
git cherry-pick -x $ancref..9f3ab14f104e81d317aa6f2cc2d8337f5e72178a

team-gateway-bot · 2024-04-15T15:11:53Z

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-12583-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-12583-to-master-to-upstream
git checkout -b cherry-pick-12583-to-master-to-upstream
ancref=$(git merge-base 1b920ca144aed6669b2b1dbe5003683b4fcd18d3 9f3ab14f104e81d317aa6f2cc2d8337f5e72178a)
git cherry-pick -x $ancref..9f3ab14f104e81d317aa6f2cc2d8337f5e72178a

team-gateway-bot · 2024-04-16T08:34:51Z

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-12583-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-12583-to-master-to-upstream
git checkout -b cherry-pick-12583-to-master-to-upstream
ancref=$(git merge-base 1b920ca144aed6669b2b1dbe5003683b4fcd18d3 9f3ab14f104e81d317aa6f2cc2d8337f5e72178a)
git cherry-pick -x $ancref..9f3ab14f104e81d317aa6f2cc2d8337f5e72178a

AntoineJac · 2024-04-16T08:49:40Z

@jschmid1 thanks could you please cherry pick and merge to EE? Thanks

dndx · 2024-04-16T09:02:59Z

@jschmid1 This squashed commit is missing the component in the scope, please watch out next time :)

This is a fixed of #12583 We need to refactor as many observability providers do not simplify ingest array. Here is a new nested format: Current model is: ```json "ai": { "mistral": { "request_total_tokens": 648, "number_of_instances": 1, "instances": [ { "meta": { "request_model": "mistral-tiny", "plugin_id": "1c783122-0223-4a03-9f1d-c71d2baab01e", "response_model": "mistral-tiny", "provider_name": "mistral" }, "usage": { "prompt_token": 392, "total_tokens": 648, "completion_token": 256 } } ], "request_prompt_tokens": 392, "request_completion_tokens": 256 }, "azure": { "request_total_tokens": 145, "number_of_instances": 1, "instances": [ { "meta": { "request_model": "gpt-35-turbo", "plugin_id": "5df193be-47a3-4f1b-8c37-37e31af0568b", "response_model": "gpt-35-turbo", "provider_name": "azure" }, "usage": { "prompt_token": 89, "total_tokens": 145, "completion_token": 56 } } ], "request_prompt_tokens": 89, "request_completion_tokens": 56 }, "cohere": { "request_total_tokens": 149, "number_of_instances": 1, "instances": [ { "meta": { "request_model": "command", "plugin_id": "30a63ab9-03c6-447a-9b25-195eb0acaeb2", "response_model": "command", "provider_name": "cohere" }, "usage": { "prompt_token": 78, "total_tokens": 149, "completion_token": 71 } } ], "request_prompt_tokens": 78, "request_completion_tokens": 71 } } ``` New model if using several plugins: ```json "ai": { "ai-request-transformer": { "meta": { "request_model": "gpt-35-turbo", "provider_name": "azure", "response_model": "gpt-35-turbo", "plugin_id": "5df193be-47a3-4f1b-8c37-37e31af0568b" }, "payload": {}, "usage": { "prompt_token": 89, "completion_token": 56, "total_tokens": 145 } }, "ai-response-transformer": { "meta": { "request_model": "mistral-tiny", "provider_name": "mistral", "response_model": "mistral-tiny", "plugin_id": "1c783122-0223-4a03-9f1d-c71d2baab01e" }, "payload": {}, "usage": { "prompt_token": 168, "completion_token": 180, "total_tokens": 348 } }, "ai-proxy": { "meta": { "request_model": "gpt-35-turbo", "provider_name": "azure", "response_model": "gpt-35-turbo", "plugin_id": "546c3856-24b3-469a-bd6c-f6083babd2cd" }, "payload": {}, "usage": { "prompt_token": 28, "completion_token": 14, "total_tokens": 42 } } }, ``` * fix analytics with new format * changelog * fix * fix json * fix test * fix test * fix test * fix(ai analytics): edit the format of the ai anayltics * fix(ai analytics): edit the format of the ai anayltics * fix: add testcase * fix: add testcase * fix: add testcase * fix lint * fix(ai-transformer): integration tests wrong response status --------- Co-authored-by: Jack Tysoe <jack.tysoe@konghq.com>

pull-request-size bot added the size/M label Feb 19, 2024

github-actions bot added plugins/ai-proxy plugins/ai-request-transformer plugins/ai-response-transformer labels Feb 19, 2024

github-actions bot assigned AntoineJac Feb 19, 2024

bungle requested a review from tysoekong February 20, 2024 07:07

AntoineJac closed this Mar 1, 2024

AntoineJac reopened this Mar 1, 2024

pull-request-size bot added size/L and removed size/M labels Mar 1, 2024

AndyZhang0707 requested a review from locao March 19, 2024 09:43

locao requested a review from gszr March 19, 2024 20:36

AntoineJac force-pushed the refactor/ai-analytics branch from a1798fc to 70c46ef Compare March 28, 2024 16:59

github-actions bot added the cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee label Mar 28, 2024

RobSerafini added this to the 3.7.0 milestone Apr 8, 2024

tysoekong force-pushed the refactor/ai-analytics branch 2 times, most recently from 935038c to bbfe32e Compare April 14, 2024 20:20

tysoekong and others added 7 commits April 14, 2024 21:39

change ai analytics

41c39da

change stat logic

9f2b5fa

refactor split

64d02d4

refactor

cbd914d

refactor

58dbd85

refactor

8ad42f0

add changelog

fa7fd04

AntoineJac and others added 3 commits April 14, 2024 21:39

update

8ae0c9a

Update handler.lua

26b5eb4

fix(rebase): solve conflicts with #12792

142eb5f

tysoekong force-pushed the refactor/ai-analytics branch from bbfe32e to 142eb5f Compare April 14, 2024 20:39

tysoekong added 2 commits April 14, 2024 23:34

feat(ai-proxy): added logging test with file-log

562bc3b

fix ai tests

96069a3

jschmid1 requested changes Apr 15, 2024

View reviewed changes

tysoekong approved these changes Apr 15, 2024

View reviewed changes

tysoekong and others added 4 commits April 15, 2024 10:28

Update spec/03-plugins/38-ai-proxy/02-openai_integration_spec.lua

e301418

Co-authored-by: Joshua Schmid <jaiks@posteo.de>

Update shared.lua

b5ac628

Update handler.lua

68bb444

Update handler.lua

9f3ab14

jschmid1 self-requested a review April 15, 2024 15:08

jschmid1 approved these changes Apr 15, 2024

View reviewed changes

jschmid1 merged commit 0db4b62 into master Apr 15, 2024
25 checks passed

jschmid1 deleted the refactor/ai-analytics branch April 15, 2024 15:10

jschmid1 added incomplete-backport and removed incomplete-backport labels Apr 15, 2024

jschmid1 changed the title ~~change ai analytics~~ refactor: change ai analytics Apr 15, 2024

AndyZhang0707 added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Apr 16, 2024

AntoineJac mentioned this pull request Apr 19, 2024

fix analytics with new format #12889

Merged

3 tasks

locao changed the title ~~refactor: change ai analytics~~ refactor(*): change ai analytics Apr 19, 2024

kikito removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Apr 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(*): change ai analytics #12583

refactor(*): change ai analytics #12583

AntoineJac commented Feb 19, 2024 •

edited by Tieske

Loading

CLAassistant commented Feb 19, 2024 •

edited

Loading

CLAassistant commented Feb 19, 2024

AntoineJac commented Mar 1, 2024 •

edited by Tieske

Loading

subnetmarco commented Mar 28, 2024

tysoekong left a comment

AntoineJac commented Apr 15, 2024

team-gateway-bot commented Apr 15, 2024

team-gateway-bot commented Apr 15, 2024

team-gateway-bot commented Apr 16, 2024

AntoineJac commented Apr 16, 2024

dndx commented Apr 16, 2024

refactor(*): change ai analytics #12583

refactor(*): change ai analytics #12583

Conversation

AntoineJac commented Feb 19, 2024 • edited by Tieske Loading

Summary

Checklist

Issue reference

CLAassistant commented Feb 19, 2024 • edited Loading

CLAassistant commented Feb 19, 2024

AntoineJac commented Mar 1, 2024 • edited by Tieske Loading

subnetmarco commented Mar 28, 2024

tysoekong left a comment

Choose a reason for hiding this comment

AntoineJac commented Apr 15, 2024

team-gateway-bot commented Apr 15, 2024

team-gateway-bot commented Apr 15, 2024

team-gateway-bot commented Apr 16, 2024

AntoineJac commented Apr 16, 2024

dndx commented Apr 16, 2024

AntoineJac commented Feb 19, 2024 •

edited by Tieske

Loading

CLAassistant commented Feb 19, 2024 •

edited

Loading

AntoineJac commented Mar 1, 2024 •

edited by Tieske

Loading