Develop ai limited preview 3 #1112

hmstepanek · 2024-03-25T16:25:59Z

Overview

Add LLM AI instrumentation for Bedrock, Langchain, and OpenAI.
Supported features:

Bedrock
- Bedrock models non-streaming:
  - amazon.titan-embed
  - amazon.titan
  - ai21.j2
  - cohere
  - anthropic.claude
  - meta.llama2
- Bedrock models streaming:
  - amazon.titan
  - cohere
  - anthropic.claude
  - meta.llama2
- Bedrock embedding models:
  - amazon.titan
  - cohere
- Bedrock via Langchain:
  - amazon.titan
  - anthropic.claude
  - meta.llama2
OpenAI
- streaming and non-streaming chat completions and embeddings
OpenAI via Langchain
- streaming and non-streaming chat completions
Langchain
- non-streaming chat completions, vectorstores, chains, agents, and tools
Note the UI does not support embeddings, vectorstores, and tools.

* Add agent instrumentation * Add pytest skip for agent tests * Change PyPDFLoader import * Remove error tests from agent

* Add support for multiple async calls * Use copy instead of deepcopy --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add LLM attr to transactions. * Remove newlines. * Add llm attribute to transaction event default attrs list. * Linting.

* Update moto decorators in tests. * Remove py27 botocore dependencies. * Drop testing for Python 3.7

* Add vectorstore error handling * Update botocore tests. #1051 * Add vector_store_id to attrs * Add vector_store_id testing * Update tox tests for botocore tests

* Add langchain tools async instrumentation. * Update run ID for errors. * Add bug fix for run_ids. * Add formatting. * [Mega-Linter] Apply linters fixes * Move metadata and tag updates. --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…wrelic-python-agent into develop-ai-limited-preview-3

* Add HanaDB and KDBAI instrumentation. * Add LLM attr to transactions. * Add testing to vectorstores. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com>

…wrelic-python-agent into develop-ai-limited-preview-3

* Mark instrumentation points for SDK (#1009) * Mark instrumentation points for SDK * Remove duplicated assertion * Fixup: assert attribute not function --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Prefix conversation id with llm (#1012) * Change conversation_id->llm.conversation_id * Fixup formatting * Add support for Meta Llama2. (#1010) * Add support for Llama2. * Fixup: lint errors * [Mega-Linter] Apply linters fixes * Trigger tests --------- Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com> * Add bedrock feedback into preview (#1030) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Bedrock Testing Infrastructure (#937) * Add AWS Bedrock testing infrastructure * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Remove OpenAI references --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Apply suggestions from code review * Remove unused import * Bedrock Sync Chat Completion Instrumentation (#953) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * TEMP * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Drop all openai refs * [Mega-Linter] Apply linters fixes * Adding response_id and response_model * Drop python 3.7 tests for Hypercorn (#954) * Apply suggestions from code review * Remove unused import --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Initial feedback commit for botocore * Feature bedrock cohere instrumentation (#955) * Add AWS Bedrock testing infrastructure * Squashed commit of the following: commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Squashed commit of the following: commit 182c7a8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Fri Oct 13 10:12:55 2023 -0700 Add request/ response IDs. commit f6d13f8 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Thu Oct 12 13:23:39 2023 -0700 Test cleanup. commit d057663 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:23:00 2023 -0700 Remove commented code. commit dd29433 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Tue Oct 10 10:19:01 2023 -0700 Add openai sync instrumentation. commit 2834663 Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Date: Mon Oct 9 17:42:05 2023 -0700 OpenAI Mock Backend (#929) * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Pin flask version for flask restx tests. (#931) * Ignore new redis methods. (#932) Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> * Remove approved paths * Update CI Image (#930) * Update available python versions in CI * Update makefile with overrides * Fix default branch detection for arm builds --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Only get package version once (#928) * Only get package version once * Add disconnect method * Add disconnect method --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Add datalib dependency for embedding testing. * Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * Add mock external openai server * Add mocked OpenAI server fixtures * Set up recorded responses. * Clean mock server to depend on http server * Linting * Remove approved paths * Add mocking for embedding endpoint * [Mega-Linter] Apply linters fixes * Add ratelimit headers * [Mega-Linter] Apply linters fixes * Add datalib dependency for embedding testing. --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> commit db63d45 Author: Uma Annamalai <uannamalai@newrelic.com> Date: Mon Oct 2 15:31:38 2023 -0700 Add OpenAI Test Infrastructure (#926) * Add openai to tox * Add OpenAI test files. * Add test functions. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com> * TEMP * Bedrock titan extraction nearly complete * Cleaning up titan bedrock implementation * TEMP * Tests for bedrock passing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> * Cleaned up titan testing Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Parametrized bedrock testing * Add support for AI21-J2 models * Change to dynamic no conversation id events * Add cohere model * Remove openai instrumentation from this branch * Remove OpenAI from newrelic/config.py --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Tim Pansino <timpansino@gmail.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Bedrock feedback w/ testing for titan and jurassic models * AWS Bedrock Embedding Instrumentation (#957) * AWS Bedrock embedding instrumentation * Correct symbol name * Add support for bedrock claude (#960) Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> * Fix merge conflicts * Combine Botocore Tests (#959) * Initial file migration * Enable DT on all span tests * Add pytest skip for older botocore versions * Fixup: app name merge conflict --------- Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Add to and move feedback tests * Handle 0.32.0.post1 version in tests (#963) * Remove response_id dependency in bedrock * Change API name * Update moto * Bedrock Error Tracing (#966) * Cache Package Version Lookups (#946) * Cache _get_package_version * Add Python 2.7 support to get_package_version caching * [Mega-Linter] Apply linters fixes * Bump tests --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> * Fix Redis Generator Methods (#947) * Fix scan_iter for redis * Replace generator methods * Update instance info instrumentation * Remove mistake from uninstrumented methods * Add skip condition to asyncio generator tests * Add skip condition to asyncio generator tests --------- Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Automatic RPM System Updates (#948) * Checkout old action * Adding RPM action * Add dry run * Incorporating action into workflow * Wire secret into custom action * Enable action * Correct action name * Fix syntax * Fix quoting issues * Drop pre-verification. Does not work on python * Fix merge artifact * Drop python 3.7 tests for Hypercorn (#954) * Fix pyenv installation for devcontainer (#936) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Remove duplicate kafka import hook (#956) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Initial bedrock error tracing commit * Handle 0.32.0.post1 version in tests (#963) * Add status code to mock bedrock server * Updating error response recording logic * Work on bedrock errror tracing * Chat completion error tracing * Adding embedding error tracing * Delete comment * Update moto --------- Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Change ids to match other tests * move message_ids declaration outside for loop * Add comment to tox.ini * Drop py27 from memcache testing. * Drop pypy27 from memcache testing. * Update flaskrestx testing #1004 * Remove tastypie 0.14.3 testing * Remove tastypie 0.14.3 testing * Remove python 3.12 support (for now) * Remove untouched files from diff list --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: Tim Pansino <timpansino@gmail.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> * Fix instrumentation for openai 1.8.0 * Add LLM attribute to transactions. (#1050) * Add LLM attr to transactions. * Remove newlines. * Add llm attribute to transaction event default attrs list. * Linting. * Remove imports for moto on py37. (#1053) * Update botocore tests. (#1051) * Update moto decorators in tests. * Remove py27 botocore dependencies. * Drop testing for Python 3.7 * Add support for streaming in openai * Add support for streaming errors * Support async generators * Add support for error during streaming * Ignore v1 tests in v0 * Refactor generator wrappers --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> Co-authored-by: hmstepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: Tim Pansino <timpansino@gmail.com> Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com> Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com> Co-authored-by: TimPansino <TimPansino@users.noreply.github.com> Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>

* Add support for streaming v1 * Fixup: coroutines return syncronously

…wrelic-python-agent into develop-ai-limited-preview-3

Merge main into develop-ai-limited-preview

* Error trace linking (#1019) * Add guid to error event & trace intrinsics w/o DT * Update tests to show guid in intrinsics sans DT * Fix cross agent DT tests and some pylint errors * Remove memcached testing for Python 2.7 * Temporarily pin hypercorn to <0.16 * Remove guid from TracedError & add to intrinsics * Rearrange intrinsics to remove redundancy * Move guid to make it agnostic to DT * Change param initialization type * Use getattr instead of calling root.guid directly * Revert cross agent tests and add guid to DT as well * Separate outside transaction error trace linking * Remove guid from DT specific attribute checks * Fix slow SQL test * Remove guid from JSON * Revert error_trace_collector validator --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Update PyPy in CI to 3.10 (#1065) * Update PyPy3.8 to PyPy3.10 in CI * Fix broken utilization test * Pin CI sha to dev image * Fix failing sklearn test * Unroll sklearn changes * Update falcon tests for 2024 SLAs * Remove pypy38 from image * Unpin dev CI image sha * Change all pypy38 to pypy310 --------- Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add guards for missing usage data in OpenAI * Remove comment * Comment out non-existent validator * Bump tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add early exit for streaming * Add vectorstores for Langchain * Trigger runs * remove py37 and change code to support py38 * Change directory of metadata.source to run on github * Fix flaskrestx testing manually * Remove py312 (for now) * Redirect instrumentation points * First round of test changes * Add test to find uninstrumented models Co-authored-by: Timothy Pansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com> * Finish reviewer updates * Add SurrealDBStore to vectorstore list * Modified mock server for OpenAI within LangChain Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> * Swap out metadata.source in test * Remove commented out code * Remove assert statement --------- Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com> Co-authored-by: Timothy Pansino <TimPansino@users.noreply.github.com> Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com> Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com>

* Add asimilarity_search support * [Mega-Linter] Apply linters fixes * Trigger tests * Cleanup code --------- Co-authored-by: lrafeei <lrafeei@users.noreply.github.com>

* Add instrumentation for sync LangChain tools. * Linter fixups. * [Mega-Linter] Apply linters fixes * Move run ID logic. * Address review feedback. * Pin OpenAI. * Fix LangChain test. * Fix tests. * Add tags check. * Combine instance and run_arg tags/ metadata. * Cleanup tests. --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add hooks and test infra for langchain * Add support for langchain chains * Use generic path for validator payload * Add async support * Record metadata and tags * Prefix metadata with metadata. * Add support for chain.run * Fix spelling typo occurr->occur Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Move tags logic up above * Remove unused response_len * Fix test name Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Refactor langchain to use shared functions * Instrument Lantern vectorstore * Test for metadata on class and in args * Unpin moto * Do not run langchain on py312 * Fix bug and remove extra lines * Fixup: spelling Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Fixup: formatting Co-authored-by: Uma Annamalai <uannamalai@newrelic.com> * Fixup: tags default to [] * Fixup lint errors newrelic/hooks/mlmodel_langchain.py:132:4: W0622: Redefining built-in 'id' (redefined-builtin) newrelic/hooks/mlmodel_langchain.py:209:4: W0622: Redefining built-in 'id' (redefined-builtin) newrelic/hooks/mlmodel_langchain.py:534:8: W1201: Use lazy % formatting in logging functions (logging-not-lazy) * Remove chain.run testing and wrapping * Unpin scipy for latest sklearn * Fix merge conflict * Fixup: re-org code for cleaner diff * Fixup: tox file comment * Fix failing tool error test --------- Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>

* Add agent instrumentation * Add pytest skip for agent tests * Change PyPDFLoader import * Remove error tests from agent

…chain-bedrock-payloads

…models

…ock-payloads Add support for langchain bedrock payloads

…models

Add tests for un-instrumented bedrock models

…dings

…lic-python-agent into support-cohere-embeddings

Add support for cohere embeddings.

umaannamalai · 2024-03-27T19:31:07Z

newrelic/api/ml_model.py

+            "id": feedback_event_id,
+            "trace_id": trace_id,
+            "rating": rating,
+            "category": category or "",


The "" can be dropped here and on the message attribute now?

umaannamalai · 2024-03-27T19:59:41Z

tests/agent_features/test_configuration.py

@@ -24,6 +24,8 @@

 import logging

+from testing_support.fixtures import override_generic_settings


was this added accidentally? it looks like it is already imported below in the file

umaannamalai · 2024-03-27T20:42:44Z

tests/agent_features/test_record_llm_feedback_event.py

+            {"type": "LlmFeedbackMessage"},
+            {
+                "id": None,
+                "category": "",


we will need to update the tests as well in conjunction with the change to the drop the or ""

…wrelic-python-agent into develop-ai-limited-preview-3

lrafeei and others added 30 commits January 24, 2024 15:39

Add langchain agent (#1043)

286a04f

* Add agent instrumentation * Add pytest skip for agent tests * Change PyPDFLoader import * Remove error tests from agent

Add HanaDB and KDBAI instrumentation.

18d896d

Add HanaDB and KDBAI instrumentation. (#1049)

433743f

Support capturing run_id during multiple async calls (#1041)

b65dc36

* Add support for multiple async calls * Use copy instead of deepcopy --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Add LLM attribute to transactions. (#1050)

8d1d19a

* Add LLM attr to transactions. * Remove newlines. * Add llm attribute to transaction event default attrs list. * Linting.

Remove imports for moto on py37. (#1053)

8a6301c

Update botocore tests. (#1051)

9ffd241

* Update moto decorators in tests. * Remove py27 botocore dependencies. * Drop testing for Python 3.7

Remove imports for moto on py37. (#1053)

02361db

Update botocore tests. (#1051)

510f747

* Update moto decorators in tests. * Remove py27 botocore dependencies. * Drop testing for Python 3.7

Langchain VectorStore error handling (#1052)

acac4b3

* Add vectorstore error handling * Update botocore tests. #1051 * Add vector_store_id to attrs * Add vector_store_id testing * Update tox tests for botocore tests

Merge branch 'develop-ai-limited-preview-3' of github.com:newrelic/ne…

2805719

…wrelic-python-agent into develop-ai-limited-preview-3

Add LLM attribute to LangChain (#1057)

4637673

* Add HanaDB and KDBAI instrumentation. * Add LLM attr to transactions. * Add testing to vectorstores. * [Mega-Linter] Apply linters fixes --------- Co-authored-by: umaannamalai <umaannamalai@users.noreply.github.com>

Merge branch 'develop-ai-limited-preview-3' of github.com:newrelic/ne…

7bf3321

…wrelic-python-agent into develop-ai-limited-preview-3

Merge branch 'main' into develop-ai-limited-preview

1584116

Openai stream v1 support (#1061)

a0a7d35

* Add support for streaming v1 * Fixup: coroutines return syncronously

Merge branch 'develop-ai-limited-preview-3' of github.com:newrelic/ne…

92b973c

…wrelic-python-agent into develop-ai-limited-preview-3

Merge pull request #1063 from newrelic/merge-main-develop-ai

56e3383

Merge main into develop-ai-limited-preview

Merge branch 'main' into develop-ai-limited-preview

c8c69c9

Add AI enabled config settings.

986cf90

Add more testing.

0c4ad7f

Fix OpenAI Missing Usage Error (#1060)

f8e8153

* Add guards for missing usage data in OpenAI * Remove comment * Comment out non-existent validator * Bump tests --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Address review feedback.

4ec5384

Instrumentation for asimilarity_search in Langchain (#1013)

374d634

* Add asimilarity_search support * [Mega-Linter] Apply linters fixes * Trigger tests * Cleanup code --------- Co-authored-by: lrafeei <lrafeei@users.noreply.github.com>

Add langchain agent (#1043)

f8df063

* Add agent instrumentation * Add pytest skip for agent tests * Change PyPDFLoader import * Remove error tests from agent

umaannamalai and others added 17 commits March 25, 2024 20:27

[Mega-Linter] Apply linters fixes

40e6aa7

Trigger tests

da511f9

Add tests for uninstrumented bedrock models

34ce84e

Merge branch 'main' into develop-ai-limited-preview-3

68f2266

Merge branch 'develop-ai-limited-preview-3' into add-support-for-lang…

4084c6f

…chain-bedrock-payloads

Merge branch 'develop-ai-limited-preview-3' into add-bedrock-missing-…

8588858

…models

Update tests/external_botocore/test_bedrock_embeddings.py

212eb55

Move langchain imports into test

76f4445

Merge pull request #1109 from newrelic/add-support-for-langchain-bedr…

680d16d

…ock-payloads Add support for langchain bedrock payloads

Merge branch 'develop-ai-limited-preview-3' into add-bedrock-missing-…

d9d6537

…models

Merge pull request #1113 from newrelic/add-bedrock-missing-models

45780e7

Add tests for un-instrumented bedrock models

Add support for cohere embeddings.

980fe04

Remove unused functions.

c35c618

Merge branch 'develop-ai-limited-preview-3' into support-cohere-embed…

53f8e70

…dings

Address review comments.

cda125a

Merge branch 'support-cohere-embeddings' of github.com:newrelic/newre…

7fffac1

…lic-python-agent into support-cohere-embeddings

Merge pull request #1115 from newrelic/support-cohere-embeddings

6f8ab45

Add support for cohere embeddings.

umaannamalai reviewed Mar 27, 2024

View reviewed changes

hmstepanek added 2 commits March 27, 2024 13:21

Merge branch 'main' into develop-ai-limited-preview-3

f689835

Fixup: code review

1216054

hmstepanek requested a review from umaannamalai March 27, 2024 20:29

umaannamalai reviewed Mar 27, 2024

View reviewed changes

umaannamalai and others added 4 commits March 27, 2024 13:50

Remove attributes from llm feedback tests.

5cce21a

Unpin s3transfer

677fbfa

Merge branch 'develop-ai-limited-preview-3' of github.com:newrelic/ne…

dc36a86

…wrelic-python-agent into develop-ai-limited-preview-3

Separate botocore langchain and botocorelatest

a21115e

umaannamalai approved these changes Mar 27, 2024

View reviewed changes

umaannamalai merged commit 0e67af5 into main Mar 27, 2024
44 of 49 checks passed

umaannamalai deleted the develop-ai-limited-preview-3 branch March 27, 2024 22:23

mergify bot removed the tests-failing label Mar 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Develop ai limited preview 3 #1112

Develop ai limited preview 3 #1112

hmstepanek commented Mar 25, 2024 •

edited

Loading

umaannamalai Mar 27, 2024

umaannamalai Mar 27, 2024

umaannamalai Mar 27, 2024

		@@ -24,6 +24,8 @@

		import logging

		from testing_support.fixtures import override_generic_settings

Develop ai limited preview 3 #1112

Develop ai limited preview 3 #1112

Conversation

hmstepanek commented Mar 25, 2024 • edited Loading

Overview

umaannamalai Mar 27, 2024

Choose a reason for hiding this comment

umaannamalai Mar 27, 2024

Choose a reason for hiding this comment

umaannamalai Mar 27, 2024

Choose a reason for hiding this comment

hmstepanek commented Mar 25, 2024 •

edited

Loading