Skip to content

Conversation

@wconti27
Copy link
Contributor

@wconti27 wconti27 commented Jun 3, 2024

This PR adds LLM Observability for Anthropic messages.

Checklist

  • Change(s) are motivated and described in the PR description
  • Testing strategy is described if automated tests are not included in the PR
  • Risks are described (performance impact, potential for breakage, maintainability)
  • Change is maintainable (easy to change, telemetry, documentation)
  • Library release note guidelines are followed or label changelog/no-changelog is set
  • Documentation is included (in-code, generated user docs, public corp docs)
  • Backport labels are set (if applicable)
  • If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Description motivates each change
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Change is maintainable (easy to change, telemetry, documentation)
  • Release note makes sense to a user of the library
  • Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

@wconti27 wconti27 requested review from a team as code owners June 3, 2024 17:02
@wconti27 wconti27 requested review from erikayasuda and removed request for a team June 3, 2024 17:02
@wconti27 wconti27 marked this pull request as draft June 3, 2024 17:03
@datadog-dd-trace-py-rkomorn
Copy link

datadog-dd-trace-py-rkomorn bot commented Jun 3, 2024

Datadog Report

Branch report: conti/add-anthropic-mlobs-methods
Commit report: 51f3fc6
Test service: dd-trace-py

✅ 0 Failed, 176975 Passed, 1900 Skipped, 11h 45m 7.67s Total duration (34m 8.47s time saved)
❄️ 6 New Flaky
⌛ 1 Performance Regression

New Flaky Tests (6)

  • test_chain_invoke_str_input - test_langchain_community.py - Last Failure

    Expand for error
     At request <Request GET /test/session/snapshot >:
        At snapshot (token='tests.contrib.langchain.test_langchain_community.test_chain_invoke'):
         - Directory: /snapshots
         - CI mode: 1
         - Trace File: /snapshots/tests.contrib.langchain.test_langchain_community.test_chain_invoke.json
         - Stats File: /snapshots/tests.contrib.langchain.test_langchain_community.test_chain_invoke_tracestats.json
         At compare of 1 expected trace(s) to 1 received trace(s):
          At trace 'langchain.request' (2 spans):
           At snapshot compare of span 'langchain.request' at position 1 in trace:
            - Expected span:
     ...
    
  • test_lcel_chain_simple - test_langchain_community.py - Last Failure

    Expand for error
     At request <Request GET /test/session/snapshot >:
        At snapshot (token='tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple'):
         - Directory: /snapshots
         - CI mode: 1
         - Trace File: /snapshots/tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple.json
         - Stats File: /snapshots/tests.contrib.langchain.test_langchain_community.test_lcel_chain_simple_tracestats.json
         At compare of 1 expected trace(s) to 1 received trace(s):
          At trace 'langchain.request' (2 spans):
           At snapshot compare of span 'langchain.request' at position 1 in trace:
            - Expected span:
     ...
    
  • test_openai_chat_model_sync_generate - test_langchain_community.py - Last Failure

    Expand for error
     At request <Request GET /test/session/snapshot >:
        At snapshot (token='tests.contrib.langchain.test_langchain_community.test_openai_chat_model_sync_generate'):
         - Directory: /snapshots
         - CI mode: 1
         - Trace File: /snapshots/tests.contrib.langchain.test_langchain_community.test_openai_chat_model_sync_generate.json
         - Stats File: /snapshots/tests.contrib.langchain.test_langchain_community.test_openai_chat_model_sync_generate_tracestats.json
         At compare of 1 expected trace(s) to 1 received trace(s):
          At trace 'langchain.request' (1 spans):
           At snapshot compare of span 'langchain.request' at position 1 in trace:
            - Expected span:
     ...
    
  • test_llmobs_anthropic_chat_model - test_langchain_llmobs.py - Last Failure

    Expand for error
     Attempted to access streaming response content, without having called \`read()\`.
    
  • test_llmobs_chain_schema_io - test_langchain_llmobs.py - Last Failure

    Expand for error
     Attempted to access streaming response content, without having called \`read()\`.
    

⌛ Performance Regressions vs Default Branch (1)

  • test_export_reset - test_http.py 6.41s (+5.35s, +504%) - Details

@wconti27 wconti27 requested a review from Yun-Kim June 4, 2024 18:29
@wconti27 wconti27 marked this pull request as ready for review June 5, 2024 13:52
@wconti27 wconti27 requested a review from a team as a code owner June 5, 2024 13:52
@wconti27 wconti27 requested a review from mabdinur June 5, 2024 13:52
@erikayasuda
Copy link
Contributor

@wconti27 LGTM just needs a release note 👍

@wconti27 wconti27 enabled auto-merge (squash) June 5, 2024 21:23
@wconti27 wconti27 merged commit 3bc3051 into main Jun 5, 2024
@wconti27 wconti27 deleted the conti/add-anthropic-mlobs-methods branch June 5, 2024 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants