Skip to content

Conversation

@liustve
Copy link
Contributor

@liustve liustve commented Dec 1, 2025

Description

Add defensive check for Bedrock responses missing expected structure. When no messageStart event with assistant role is received, the response may lack the standard output.message format, causing the occasional KeyError exceptions.

Fixes #3958

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

I was unable to reproduce the exact scenario that would trigger this bug , this safety check would fix this without affecting normal instrumentation behavior.

Does This PR Require a Core Repo Change?

  • [x ] No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@liustve liustve requested a review from a team as a code owner December 1, 2025 22:57
@liustve liustve force-pushed the fix-bedrock-converse-stream-bug branch from 65e10cb to fde933a Compare December 1, 2025 23:00
@liustve liustve force-pushed the fix-bedrock-converse-stream-bug branch from 05bcdf9 to 0861f9f Compare December 1, 2025 23:03
…ntelemetry/instrumentation/botocore/extensions/bedrock_utils.py
@xrmx
Copy link
Contributor

xrmx commented Dec 2, 2025

@liustve Please fix the CI failures and next time please run the tests locally :)

@xrmx xrmx moved this to Approved PRs that need fixes in @xrmx's Python PR digest Dec 2, 2025
@xrmx
Copy link
Contributor

xrmx commented Dec 2, 2025

@liustve I don't think you pushed a fix for the test:

_____________ test_converse_stream_with_missing_output_in_response _____________

    def test_converse_stream_with_missing_output_in_response():
        """Test that converse stream handles malformed response missing output key."""
    
        def stream_done_callback(response, ended):
            pass
    
>       wrapper = ConverseStreamWrapper(
            stream=mock.MagicMock(),
            stream_done_callback=stream_done_callback,
            model_id="amazon.nova-micro-v1:0",
        )
E       TypeError: ConverseStreamWrapper.__init__() got an unexpected keyword argument 'model_id'

instrumentation/opentelemetry-instrumentation-botocore/tests/test_botocore_bedrock.py:3061: TypeError

@liustve
Copy link
Contributor Author

liustve commented Dec 2, 2025

@liustve I don't think you pushed a fix for the test:

_____________ test_converse_stream_with_missing_output_in_response _____________

    def test_converse_stream_with_missing_output_in_response():
        """Test that converse stream handles malformed response missing output key."""
    
        def stream_done_callback(response, ended):
            pass
    
>       wrapper = ConverseStreamWrapper(
            stream=mock.MagicMock(),
            stream_done_callback=stream_done_callback,
            model_id="amazon.nova-micro-v1:0",
        )
E       TypeError: ConverseStreamWrapper.__init__() got an unexpected keyword argument 'model_id'

instrumentation/opentelemetry-instrumentation-botocore/tests/test_botocore_bedrock.py:3061: TypeError

Apologies, had to fix a few things in my local dev environment, for some reason the tests were using another version of OTel botocore.

Also revised the test to better capture the intended behavior.

@liustve
Copy link
Contributor Author

liustve commented Dec 2, 2025

@xrmx added another defensive check for orig_message["content"] as well. Just in case.

liustve added a commit to aws-observability/aws-otel-python-instrumentation that referenced this pull request Dec 2, 2025
*Description of changes:*

Ports the changes from
open-telemetry/opentelemetry-python-contrib#3990

Add defensive check for Bedrock responses missing expected structure.
When no messageStart event with assistant role is received, the response
may lack the standard output.message format, causing the occasional
KeyError exceptions.

For testing, I was unable to reproduce the exact scenario that would
trigger this bug, however, this safety check would fix this issue
without affecting normal instrumentation behavior.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
@xrmx xrmx merged commit be8dfdc into open-telemetry:main Dec 3, 2025
647 checks passed
@github-project-automation github-project-automation bot moved this from Approved PRs that need fixes to Done in @xrmx's Python PR digest Dec 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

bedrock_utils.py throwing an error on

2 participants