Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bedrock contract tests. #227

Merged
merged 9 commits into from
Aug 16, 2024
Merged

Add bedrock contract tests. #227

merged 9 commits into from
Aug 16, 2024

Conversation

zzhlogin
Copy link
Contributor

@zzhlogin zzhlogin commented Jul 11, 2024

The PR is a follow up on bedrock service support PR:
#209

We add contract tests for following Bedrock services that covers all resource attributes we newly support:

  1. Bedrock API: GetGuardrail
  2. BedrockAgent APIs: GetAgent, GetDataSource, GetKnowledgeBase
  3. BedrockRuntime API: InvokeModel
  4. BedrockAgentRuntime API: InvokeAgent

Upgrade botocore and boto3 to the latest version 1.34.143 so that to support Bedrock services API calls.
Upgrade localstack/localstack image to the latest version 3.5.0 so resolve the SQS API call issue using localstack/localstack:2.0.1 with new version of boto3: localstack/localstack#9610

Contract test limitation:
The contract tests in current repo is using LocalStackContainer to serve AWS SDK service calls. But it doesn’t has bedrock related service support (This is the full service list it support.). In this case, no matter which bedrock API we call in contract test, the response will always be 4XX.

As a workaround, we inject inject_200_success and inject_500_error directly into the API call to make sure we receive http response with expected status code and attributes.

_assert_semantic_conventions_span_attributes function change:
In _assert_semantic_conventions_span_attributes function it is checking the input service equals to rpc.service, however, we pass the input service with "remote_service", where there is mismatch for example for Bedrock Agent Runtime: we have
rpc_service="Bedrock Agent Runtime", remote_service="AWS::Bedrock".
Thus we change to use rpc_service if it is provided by:

kwargs.get("rpc_service") if "rpc_service" in kwargs else kwargs.get("remote_service").split("::")[-1],

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@zzhlogin zzhlogin requested a review from a team as a code owner July 11, 2024 21:45
Copy link
Member

@mxiamxia mxiamxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general, the pr looks good to me. the tests covers the new gen ai attributes will be released.

mxiamxia
mxiamxia previously approved these changes Jul 26, 2024
@zzhlogin zzhlogin merged commit 0d05ca7 into main Aug 16, 2024
11 checks passed
@zzhlogin zzhlogin deleted the bedrock-contract-test branch August 16, 2024 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants