[FEATURE] Add Anthropic LLM support via `anthropic` Python SDK #332

the-praxs · 2024-08-05T17:57:15Z

📥 Pull Request

📘 Description
This PR adds functions in the llm_tracker.py file for the asynchronous and synchronous Anthropic SDK support. An example notebook anthropic_example under examples/anthropic is also added to demo the integration.

🎯 Goal
To extend the AgentOps capability to the Anthropic LLMs.

🔍 Additional Context
Tool Calling in Anthropic Python SDK seems flaky and might not result in correct responses,

🧪 Testing
Tested using the provided example notebook.

Thank you for your contribution to Agentops!

gitguardian · 2024-08-10T15:48:45Z

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

codecov · 2024-08-10T15:52:51Z

Codecov Report

Attention: Patch coverage is 0% with 668 lines in your changes missing coverage. Please review.

Files	Patch %	Lines
agentops/llm_tracker.py	0.00%	668 Missing ⚠️

Flag	Coverage Δ
unittests	`34.85% <0.00%> (-10.75%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
agentops/event.py	`92.00% <ø> (ø)`
agentops/llm_tracker.py	`0.00% <0.00%> (ø)`

the-praxs · 2024-08-11T13:48:34Z

@bboynton97 @areibman This is ready for review!

areibman

Really great PR! I left a few comments. Namely, Async isn't parsing data as we'd expect. Otherwise looks good

areibman · 2024-08-11T20:20:35Z

examples/anthropic/anthropic_example.ipynb

+    "for event in stream:\n",
+    "    if event.type == \"content_block_delta\":\n",
+    "        if event.delta.type == \"text\":\n",
+    "            response += event.delta.text\n",
+    "    elif event.type == \"message_stop\":\n",
+    "        print(\"\\n\")\n",
+    "        print(response)\n",
+    "        print(\"\\n\")"


The second example doesn't print anything for some reason

However, it does get recorded in AgentOps:

I re-ran the code and found out the tool calling isn't doing what it's expected, in this case, performing a web search). This is why it's only printing the "thinking" part but not the "answer". I have 2 solutions in mind for a fix -

Precisely describe the web search in the tool

Use the anthropic-tools package which is in alpha stage

I will be testing both of these and push a fix soon.

I tested with both of these and so far no luck since there is no innate web search support for the models. However, another solution would be to use the anthropic-tools package with a custom Python class that uses another package to perform web searches, getting us the results we want. Or we could skip such questions and ask something else in the prompt.

Please let me know what are your thoughts on this.

agentops/llm_tracker.py

examples/anthropic/anthropic_example.ipynb

areibman · 2024-08-16T01:16:39Z

Unfortunate timing for this PR. @bboynton97 is currently splitting llm_tracker.py into multiple files for readability. When that merge goes through, we'll make some changes to make this one work

the-praxs · 2024-08-16T03:55:40Z

Unfortunate timing for this PR. @bboynton97 is currently splitting llm_tracker.py into multiple files for readability. When that merge goes through, we'll make some changes to make this one work

I'd like to help, would be good for me to understand the working of the software in detail and I can use that information to draft a contributing guidelines document and/or observe where tests are needed.

areibman · 2024-08-16T04:51:16Z

Unfortunate timing for this PR. @bboynton97 is currently splitting llm_tracker.py into multiple files for readability. When that merge goes through, we'll make some changes to make this one work

I'd like to help, would be good for me to understand the working of the software in detail and I can use that information to draft a contributing guidelines document and/or observe where tests are needed.

This is the PR in question: #346

From your end, this would just mean splitting the Anthropic handlers into its own file. The tricky part is just merging those changes from that branch into yours.

areibman · 2024-08-16T04:51:49Z

@HowieG Can you add the Anthropic notebook to the smoke test suite?

the-praxs · 2024-08-16T04:54:39Z

Unfortunate timing for this PR. @bboynton97 is currently splitting llm_tracker.py into multiple files for readability. When that merge goes through, we'll make some changes to make this one work

I'd like to help, would be good for me to understand the working of the software in detail and I can use that information to draft a contributing guidelines document and/or observe where tests are needed.

This is the PR in question: #346

From your end, this would just mean splitting the Anthropic handlers into its own file. The tricky part is just merging those changes from that branch into yours.

Cool, I will look into it and find a solution.

HowieG · 2024-08-16T08:45:54Z

@HowieG Can you add the Anthropic notebook to the smoke test suite?

Any notebook added to tests or examples folders will be smoke tested automatically. Except any we've excluded

the-praxs · 2024-08-17T11:57:31Z

Unfortunate timing for this PR. @bboynton97 is currently splitting llm_tracker.py into multiple files for readability. When that merge goes through, we'll make some changes to make this one work

I'd like to help, would be good for me to understand the working of the software in detail and I can use that information to draft a contributing guidelines document and/or observe where tests are needed.

This is the PR in question: #346
From your end, this would just mean splitting the Anthropic handlers into its own file. The tricky part is just merging those changes from that branch into yours.

Cool, I will look into it and find a solution.

I rebased and refactored the code. I believe that this PR should merge to llm-handler-refactor if that's where the changes are right now.

bboynton97 · 2024-08-19T22:19:03Z

hey @the-praxs! thank you for reworking this, it looks fantastic! i'm going to go ahead and merge this into the llm-handler-refactor like you set it up to do and make sure you get credit on the subsequent PR as well :)

thank you for your work here! its very appreciated

* separated llm tracker by model * llm refactor progress * working for openai * groq provider * cohere provider * ollama provider * remove support for openai v0 * cohere support * test import fix * test import fix * groq test fix * ollama tests * litellm tests * dont import litellm * cohere fixes and tests * oai version <0.1 better deprecation warning * [FEATURE] Add Anthropic LLM support via `anthropic` Python SDK (#332) * fix typo * add anthropic support * add example file * fixed linting using black * add time travel support for anthropic * linting * fix kwargs key to get prompt * fix for extracting tokens from Message.usage * minor fix for output tokens * remove anthropic example python file * fix completions not show in session * some more fixes and cleanup * add Message object to pydantic models * fix typo * overhaul anthropic code * linting * add anthropic example notebook * linting * added readme examples * fix incorrect attribute access for the model content * add async example * refactor code * fix function name * linting * add provider tests --------- Co-authored-by: reibs <areibman@gmail.com> * added undo to canaries * added anthropic tests * undo instrumenting for litellm * cohere considerations --------- Co-authored-by: Pratyush Shukla <ps4534@nyu.edu> Co-authored-by: reibs <areibman@gmail.com>

the-praxs marked this pull request as ready for review August 10, 2024 22:55

areibman requested changes Aug 11, 2024

View reviewed changes

the-praxs and others added 19 commits August 17, 2024 16:41

fix typo

34ed33d

add anthropic support

0a0a594

add example file

4576fec

fixed linting using black

f57b1cf

add time travel support for anthropic

86c56c9

linting

40aeb1a

fix kwargs key to get prompt

519e733

fix for extracting tokens from Message.usage

9eaa73e

minor fix for output tokens

99226d9

remove anthropic example python file

063ad2b

fix completions not show in session

d0f38ac

some more fixes and cleanup

14a1739

add Message object to pydantic models

130d9c2

fix typo

a911bec

overhaul anthropic code

013d35b

linting

959d971

add anthropic example notebook

198a09f

linting

72fb343

added readme examples

7e67af8

the-praxs added 2 commits August 17, 2024 16:42

fix incorrect attribute access for the model content

4217d22

add async example

cde0820

the-praxs force-pushed the anthropic-ops branch from 44428ff to cde0820 Compare August 17, 2024 11:49

the-praxs changed the base branch from main to llm-handler-refactor August 17, 2024 11:50

refactor code

7a17a4c

the-praxs added 4 commits August 17, 2024 17:31

fix function name

6bc844f

linting

4a8e23d

Merge branch 'llm-handler-refactor' into anthropic-ops

58a9f24

add provider tests

0f403c4

the-praxs changed the title ~~[FEATURE] Add Anthropic LLM support via Anthropic Python SDK~~ [FEATURE] Add Anthropic LLM support via anthropic Python SDK Aug 18, 2024

bboynton97 merged commit 47633ec into AgentOps-AI:llm-handler-refactor Aug 19, 2024
3 of 4 checks passed

the-praxs deleted the anthropic-ops branch August 20, 2024 02:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Add Anthropic LLM support via `anthropic` Python SDK #332

[FEATURE] Add Anthropic LLM support via `anthropic` Python SDK #332

the-praxs commented Aug 5, 2024 •

edited

Loading

gitguardian bot commented Aug 10, 2024 •

edited

Loading

codecov bot commented Aug 10, 2024 •

edited

Loading

the-praxs commented Aug 11, 2024

areibman left a comment

areibman Aug 11, 2024

areibman Aug 11, 2024

the-praxs Aug 12, 2024 •

edited

Loading

the-praxs Aug 14, 2024 •

edited

Loading

areibman commented Aug 16, 2024

the-praxs commented Aug 16, 2024

areibman commented Aug 16, 2024

areibman commented Aug 16, 2024

the-praxs commented Aug 16, 2024

HowieG commented Aug 16, 2024

the-praxs commented Aug 17, 2024

bboynton97 commented Aug 19, 2024

[FEATURE] Add Anthropic LLM support via anthropic Python SDK #332

[FEATURE] Add Anthropic LLM support via anthropic Python SDK #332

Conversation

the-praxs commented Aug 5, 2024 • edited Loading

📥 Pull Request

gitguardian bot commented Aug 10, 2024 • edited Loading

️✅ There are no secrets present in this pull request anymore.

codecov bot commented Aug 10, 2024 • edited Loading

Codecov Report

the-praxs commented Aug 11, 2024

areibman left a comment

Choose a reason for hiding this comment

areibman Aug 11, 2024

Choose a reason for hiding this comment

areibman Aug 11, 2024

Choose a reason for hiding this comment

the-praxs Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

the-praxs Aug 14, 2024 • edited Loading

Choose a reason for hiding this comment

areibman commented Aug 16, 2024

the-praxs commented Aug 16, 2024

areibman commented Aug 16, 2024

areibman commented Aug 16, 2024

the-praxs commented Aug 16, 2024

HowieG commented Aug 16, 2024

the-praxs commented Aug 17, 2024

bboynton97 commented Aug 19, 2024

[FEATURE] Add Anthropic LLM support via `anthropic` Python SDK #332

[FEATURE] Add Anthropic LLM support via `anthropic` Python SDK #332

the-praxs commented Aug 5, 2024 •

edited

Loading

gitguardian bot commented Aug 10, 2024 •

edited

Loading

codecov bot commented Aug 10, 2024 •

edited

Loading

the-praxs Aug 12, 2024 •

edited

Loading

the-praxs Aug 14, 2024 •

edited

Loading