Skip to content

Commit

Permalink
Add bedrock feedback into preview (#1030)
Browse files Browse the repository at this point in the history
* Add AWS Bedrock testing infrastructure

* Squashed commit of the following:

commit 2834663
Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Date:   Mon Oct 9 17:42:05 2023 -0700

    OpenAI Mock Backend (#929)

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Pin flask version for flask restx tests. (#931)

    * Ignore new redis methods. (#932)

    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

    * Remove approved paths

    * Update CI Image (#930)

    * Update available python versions in CI

    * Update makefile with overrides

    * Fix default branch detection for arm builds

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Only get package version once (#928)

    * Only get package version once

    * Add disconnect method

    * Add disconnect method

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add datalib dependency for embedding testing.

    * Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Remove approved paths

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Add datalib dependency for embedding testing.

    ---------

    Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
    Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

commit db63d45
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Mon Oct 2 15:31:38 2023 -0700

    Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

* Squashed commit of the following:

commit 182c7a8
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Fri Oct 13 10:12:55 2023 -0700

    Add request/ response IDs.

commit f6d13f8
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Thu Oct 12 13:23:39 2023 -0700

    Test cleanup.

commit d057663
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Tue Oct 10 10:23:00 2023 -0700

    Remove commented code.

commit dd29433
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Tue Oct 10 10:19:01 2023 -0700

    Add openai sync instrumentation.

commit 2834663
Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Date:   Mon Oct 9 17:42:05 2023 -0700

    OpenAI Mock Backend (#929)

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Pin flask version for flask restx tests. (#931)

    * Ignore new redis methods. (#932)

    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

    * Remove approved paths

    * Update CI Image (#930)

    * Update available python versions in CI

    * Update makefile with overrides

    * Fix default branch detection for arm builds

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Only get package version once (#928)

    * Only get package version once

    * Add disconnect method

    * Add disconnect method

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add datalib dependency for embedding testing.

    * Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Remove approved paths

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Add datalib dependency for embedding testing.

    ---------

    Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
    Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

commit db63d45
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Mon Oct 2 15:31:38 2023 -0700

    Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

* TEMP

* Bedrock titan extraction nearly complete

* Bedrock Testing Infrastructure (#937)

* Add AWS Bedrock testing infrastructure

* Cache Package Version Lookups (#946)

* Cache _get_package_version

* Add Python 2.7 support to get_package_version caching

* [Mega-Linter] Apply linters fixes

* Bump tests

---------

Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>

* Fix Redis Generator Methods (#947)

* Fix scan_iter for redis

* Replace generator methods

* Update instance info instrumentation

* Remove mistake from uninstrumented methods

* Add skip condition to asyncio generator tests

* Add skip condition to asyncio generator tests

---------

Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Automatic RPM System Updates (#948)

* Checkout old action

* Adding RPM action

* Add dry run

* Incorporating action into workflow

* Wire secret into custom action

* Enable action

* Correct action name

* Fix syntax

* Fix quoting issues

* Drop pre-verification. Does not work on python

* Fix merge artifact

* Remove OpenAI references

---------

Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Cleaning up titan bedrock implementation

* TEMP

* Tests for bedrock passing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>

* Cleaned up titan testing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Parametrized bedrock testing

* Add support for AI21-J2 models

* Change to dynamic no conversation id events

* Drop all openai refs

* [Mega-Linter] Apply linters fixes

* Adding response_id and response_model

* Apply suggestions from code review

* Remove unused import

* Bedrock Sync Chat Completion Instrumentation (#953)

* Add AWS Bedrock testing infrastructure

* Squashed commit of the following:

commit 2834663
Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Date:   Mon Oct 9 17:42:05 2023 -0700

    OpenAI Mock Backend (#929)

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Pin flask version for flask restx tests. (#931)

    * Ignore new redis methods. (#932)

    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

    * Remove approved paths

    * Update CI Image (#930)

    * Update available python versions in CI

    * Update makefile with overrides

    * Fix default branch detection for arm builds

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Only get package version once (#928)

    * Only get package version once

    * Add disconnect method

    * Add disconnect method

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add datalib dependency for embedding testing.

    * Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Remove approved paths

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Add datalib dependency for embedding testing.

    ---------

    Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
    Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

commit db63d45
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Mon Oct 2 15:31:38 2023 -0700

    Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

* Squashed commit of the following:

commit 182c7a8
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Fri Oct 13 10:12:55 2023 -0700

    Add request/ response IDs.

commit f6d13f8
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Thu Oct 12 13:23:39 2023 -0700

    Test cleanup.

commit d057663
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Tue Oct 10 10:23:00 2023 -0700

    Remove commented code.

commit dd29433
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Tue Oct 10 10:19:01 2023 -0700

    Add openai sync instrumentation.

commit 2834663
Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Date:   Mon Oct 9 17:42:05 2023 -0700

    OpenAI Mock Backend (#929)

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Pin flask version for flask restx tests. (#931)

    * Ignore new redis methods. (#932)

    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

    * Remove approved paths

    * Update CI Image (#930)

    * Update available python versions in CI

    * Update makefile with overrides

    * Fix default branch detection for arm builds

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Only get package version once (#928)

    * Only get package version once

    * Add disconnect method

    * Add disconnect method

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add datalib dependency for embedding testing.

    * Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Remove approved paths

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Add datalib dependency for embedding testing.

    ---------

    Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
    Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

commit db63d45
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Mon Oct 2 15:31:38 2023 -0700

    Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

* Cache Package Version Lookups (#946)

* Cache _get_package_version

* Add Python 2.7 support to get_package_version caching

* [Mega-Linter] Apply linters fixes

* Bump tests

---------

Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>

* Fix Redis Generator Methods (#947)

* Fix scan_iter for redis

* Replace generator methods

* Update instance info instrumentation

* Remove mistake from uninstrumented methods

* Add skip condition to asyncio generator tests

* Add skip condition to asyncio generator tests

---------

Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* TEMP

* Automatic RPM System Updates (#948)

* Checkout old action

* Adding RPM action

* Add dry run

* Incorporating action into workflow

* Wire secret into custom action

* Enable action

* Correct action name

* Fix syntax

* Fix quoting issues

* Drop pre-verification. Does not work on python

* Fix merge artifact

* Bedrock titan extraction nearly complete

* Cleaning up titan bedrock implementation

* TEMP

* Tests for bedrock passing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>

* Cleaned up titan testing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Parametrized bedrock testing

* Add support for AI21-J2 models

* Change to dynamic no conversation id events

* Drop all openai refs

* [Mega-Linter] Apply linters fixes

* Adding response_id and response_model

* Drop python 3.7 tests for Hypercorn (#954)

* Apply suggestions from code review

* Remove unused import

---------

Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

* Initial feedback commit for botocore

* Feature bedrock cohere instrumentation (#955)

* Add AWS Bedrock testing infrastructure

* Squashed commit of the following:

commit 2834663
Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Date:   Mon Oct 9 17:42:05 2023 -0700

    OpenAI Mock Backend (#929)

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Pin flask version for flask restx tests. (#931)

    * Ignore new redis methods. (#932)

    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

    * Remove approved paths

    * Update CI Image (#930)

    * Update available python versions in CI

    * Update makefile with overrides

    * Fix default branch detection for arm builds

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Only get package version once (#928)

    * Only get package version once

    * Add disconnect method

    * Add disconnect method

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add datalib dependency for embedding testing.

    * Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Remove approved paths

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Add datalib dependency for embedding testing.

    ---------

    Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
    Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

commit db63d45
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Mon Oct 2 15:31:38 2023 -0700

    Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

* Squashed commit of the following:

commit 182c7a8
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Fri Oct 13 10:12:55 2023 -0700

    Add request/ response IDs.

commit f6d13f8
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Thu Oct 12 13:23:39 2023 -0700

    Test cleanup.

commit d057663
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Tue Oct 10 10:23:00 2023 -0700

    Remove commented code.

commit dd29433
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Tue Oct 10 10:19:01 2023 -0700

    Add openai sync instrumentation.

commit 2834663
Author: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Date:   Mon Oct 9 17:42:05 2023 -0700

    OpenAI Mock Backend (#929)

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Pin flask version for flask restx tests. (#931)

    * Ignore new redis methods. (#932)

    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>

    * Remove approved paths

    * Update CI Image (#930)

    * Update available python versions in CI

    * Update makefile with overrides

    * Fix default branch detection for arm builds

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Only get package version once (#928)

    * Only get package version once

    * Add disconnect method

    * Add disconnect method

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

    * Add datalib dependency for embedding testing.

    * Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

    * Add mock external openai server

    * Add mocked OpenAI server fixtures

    * Set up recorded responses.

    * Clean mock server to depend on http server

    * Linting

    * Remove approved paths

    * Add mocking for embedding endpoint

    * [Mega-Linter] Apply linters fixes

    * Add ratelimit headers

    * [Mega-Linter] Apply linters fixes

    * Add datalib dependency for embedding testing.

    ---------

    Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
    Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
    Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

commit db63d45
Author: Uma Annamalai <uannamalai@newrelic.com>
Date:   Mon Oct 2 15:31:38 2023 -0700

    Add OpenAI Test Infrastructure (#926)

    * Add openai to tox

    * Add OpenAI test files.

    * Add test functions.

    * [Mega-Linter] Apply linters fixes

    ---------

    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <mergify[bot]@users.noreply.github.com>

* TEMP

* Bedrock titan extraction nearly complete

* Cleaning up titan bedrock implementation

* TEMP

* Tests for bedrock passing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>

* Cleaned up titan testing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Parametrized bedrock testing

* Add support for AI21-J2 models

* Change to dynamic no conversation id events

* Add cohere model

* Remove openai instrumentation from this branch

* Remove OpenAI from newrelic/config.py

---------

Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
Co-authored-by: Tim Pansino <timpansino@gmail.com>
Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Bedrock feedback w/ testing for titan and jurassic models

* AWS Bedrock Embedding Instrumentation (#957)

* AWS Bedrock embedding instrumentation

* Correct symbol name

* Add support for bedrock claude (#960)

Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>

* Fix merge conflicts

* Combine Botocore Tests (#959)

* Initial file migration

* Enable DT on all span tests

* Add pytest skip for older botocore versions

* Fixup: app name merge conflict

---------

Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>

* Add to and move feedback tests

* Handle 0.32.0.post1 version in tests (#963)

* Remove response_id dependency in bedrock

* Change API name

* Update moto

* Bedrock Error Tracing (#966)

* Cache Package Version Lookups (#946)

* Cache _get_package_version

* Add Python 2.7 support to get_package_version caching

* [Mega-Linter] Apply linters fixes

* Bump tests

---------

Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>

* Fix Redis Generator Methods (#947)

* Fix scan_iter for redis

* Replace generator methods

* Update instance info instrumentation

* Remove mistake from uninstrumented methods

* Add skip condition to asyncio generator tests

* Add skip condition to asyncio generator tests

---------

Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Automatic RPM System Updates (#948)

* Checkout old action

* Adding RPM action

* Add dry run

* Incorporating action into workflow

* Wire secret into custom action

* Enable action

* Correct action name

* Fix syntax

* Fix quoting issues

* Drop pre-verification. Does not work on python

* Fix merge artifact

* Drop python 3.7 tests for Hypercorn (#954)

* Fix pyenv installation for devcontainer (#936)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Remove duplicate kafka import hook (#956)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Initial bedrock error tracing commit

* Handle 0.32.0.post1 version in tests (#963)

* Add status code to mock bedrock server

* Updating error response recording logic

* Work on bedrock errror tracing

* Chat completion error tracing

* Adding embedding error tracing

* Delete comment

* Update moto

---------

Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <lrafeei@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <84813886+lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>

* Change ids to match other tests

* move message_ids declaration outside for loop

* Add comment to tox.ini

* Drop py27 from memcache testing.

* Drop pypy27 from memcache testing.

* Update flaskrestx testing #1004

* Remove tastypie 0.14.3 testing

* Remove tastypie 0.14.3 testing

* Remove python 3.12 support (for now)

* Remove untouched files from diff list

---------

Co-authored-by: Uma Annamalai <uannamalai@newrelic.com>
Co-authored-by: Tim Pansino <timpansino@gmail.com>
Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Co-authored-by: SlavaSkvortsov <29122694+SlavaSkvortsov@users.noreply.github.com>
Co-authored-by: TimPansino <TimPansino@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
  • Loading branch information
10 people committed Feb 13, 2024
1 parent dbcbca5 commit 3f82021
Show file tree
Hide file tree
Showing 6 changed files with 119 additions and 7 deletions.
9 changes: 6 additions & 3 deletions newrelic/api/ml_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,15 @@ def wrap_mlmodel(model, name=None, version=None, feature_names=None, label_names

def get_llm_message_ids(response_id=None):
transaction = current_transaction()
if response_id and transaction:
if transaction:
nr_message_ids = getattr(transaction, "_nr_message_ids", {})
message_id_info = nr_message_ids.pop(response_id, ())
message_id_info = (
nr_message_ids.pop("bedrock_key", ()) if not response_id else nr_message_ids.pop(response_id, ())
)

if not message_id_info:
warnings.warn("No message ids found for %s" % response_id)
response_id_warning = "." if not response_id else " for %s." % response_id
warnings.warn("No message ids found%s" % response_id_warning)
return []

conversation_id, request_id, ids = message_id_info
Expand Down
9 changes: 8 additions & 1 deletion newrelic/hooks/external_botocore.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ def create_chat_completion_message_event(
if not transaction:
return

message_ids = []
for index, message in enumerate(input_message_list):
if response_id:
id_ = "%s-%d" % (response_id, index) # Response ID was set, append message index to it.
Expand Down Expand Up @@ -128,6 +129,7 @@ def create_chat_completion_message_event(
id_ = "%s-%d" % (response_id, index) # Response ID was set, append message index to it.
else:
id_ = str(uuid.uuid4()) # No response IDs, use random UUID
message_ids.append(id_)

chat_completion_message_dict = {
"id": id_,
Expand All @@ -147,6 +149,7 @@ def create_chat_completion_message_event(
"is_response": True,
}
transaction.record_custom_event("LlmChatCompletionMessage", chat_completion_message_dict)
return (conversation_id, request_id, message_ids)


def extract_bedrock_titan_text_model(request_body, response_body=None):
Expand Down Expand Up @@ -577,7 +580,7 @@ def handle_chat_completion_event(

transaction.record_custom_event("LlmChatCompletionSummary", chat_completion_summary_dict)

create_chat_completion_message_event(
message_ids = create_chat_completion_message_event(
transaction=transaction,
app_name=settings.app_name,
input_message_list=input_message_list,
Expand All @@ -591,6 +594,10 @@ def handle_chat_completion_event(
response_id=response_id,
)

if not hasattr(transaction, "_nr_message_ids"):
transaction._nr_message_ids = {}
transaction._nr_message_ids["bedrock_key"] = message_ids


CUSTOM_TRACE_POINTS = {
("sns", "publish"): message_trace("SNS", "Produce", "Topic", extract(("TopicArn", "TargetArn"), "PhoneNumber")),
Expand Down
87 changes: 87 additions & 0 deletions tests/external_botocore/_test_bedrock_chat_completion.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2010 New Relic, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

chat_completion_payload_templates = {
"amazon.titan-text-express-v1": '{ "inputText": "%s", "textGenerationConfig": {"temperature": %f, "maxTokenCount": %d }}',
"ai21.j2-mid-v1": '{"prompt": "%s", "temperature": %f, "maxTokens": %d}',
Expand All @@ -6,6 +20,79 @@
"meta.llama2-13b-chat-v1": '{"prompt": "%s", "temperature": %f, "max_gen_len": %d}',
}

chat_completion_get_llm_message_ids = {
"amazon.titan-text-express-v1": {
"bedrock_key": [
{
"conversation_id": "my-awesome-id",
"request_id": "03524118-8d77-430f-9e08-63b5c03a40cf",
"message_id": None, # UUID that varies with each run
},
{
"conversation_id": "my-awesome-id",
"request_id": "03524118-8d77-430f-9e08-63b5c03a40cf",
"message_id": None, # UUID that varies with each run
},
]
},
"ai21.j2-mid-v1": {
"bedrock_key": [
{
"conversation_id": "my-awesome-id",
"request_id": "c863d9fc-888b-421c-a175-ac5256baec62",
"message_id": "1234-0",
},
{
"conversation_id": "my-awesome-id",
"request_id": "c863d9fc-888b-421c-a175-ac5256baec62",
"message_id": "1234-1",
},
]
},
"anthropic.claude-instant-v1": {
"bedrock_key": [
{
"conversation_id": "my-awesome-id",
"request_id": "7b0b37c6-85fb-4664-8f5b-361ca7b1aa18",
"message_id": None, # UUID that varies with each run
},
{
"conversation_id": "my-awesome-id",
"request_id": "7b0b37c6-85fb-4664-8f5b-361ca7b1aa18",
"message_id": None, # UUID that varies with each run
},
]
},
"cohere.command-text-v14": {
"bedrock_key": [
{
"conversation_id": "my-awesome-id",
"request_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97",
"message_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97-0",
},
{
"conversation_id": "my-awesome-id",
"request_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97",
"message_id": "e77422c8-fbbf-4e17-afeb-c758425c9f97-1",
},
]
},
"meta.llama2-13b-chat-v1": {
"bedrock_key": [
{
"conversation_id": "my-awesome-id",
"request_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a",
"message_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a-0",
},
{
"conversation_id": "my-awesome-id",
"request_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a",
"message_id": "9a64cdb0-3e82-41c7-873a-c12a77e0143a-1",
},
]
},
}

chat_completion_expected_events = {
"amazon.titan-text-express-v1": [
(
Expand Down
18 changes: 16 additions & 2 deletions tests/external_botocore/_test_bedrock_embeddings.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2010 New Relic, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

embedding_payload_templates = {
"amazon.titan-embed-text-v1": '{ "inputText": "%s" }',
"amazon.titan-embed-g1-text-02": '{ "inputText": "%s" }',
Expand Down Expand Up @@ -68,7 +82,7 @@
"request_id": "",
"vendor": "bedrock",
"ingest_source": "Python",
"error": True
"error": True,
},
),
],
Expand All @@ -89,7 +103,7 @@
"request_id": "",
"vendor": "bedrock",
"ingest_source": "Python",
"error": True
"error": True,
},
),
],
Expand Down
2 changes: 1 addition & 1 deletion tests/external_botocore/test_bedrock_embeddings.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
import pytest
from _test_bedrock_embeddings import (
embedding_expected_client_errors,
embedding_expected_events,
embedding_expected_error_events,
embedding_expected_events,
embedding_payload_templates,
)
from conftest import BOTOCORE_VERSION
Expand Down
1 change: 1 addition & 0 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ deps =
component_flask_rest: flask-restful
component_flask_rest: jinja2
component_flask_rest: itsdangerous
component_flask_rest-flaskrestxlatest: flask
component_flask_rest-flaskrestxlatest: flask-restx
component_flask_rest-flaskrestxlatest: flask
; flask-restx only supports Flask v3 after flask-restx v1.3.0
Expand Down

0 comments on commit 3f82021

Please sign in to comment.