Skip to content

define client each handler invocation#4662

Merged
ellenmuller merged 3 commits into
mainfrom
em-reinitialise-bedrock-client
Mar 11, 2026
Merged

define client each handler invocation#4662
ellenmuller merged 3 commits into
mainfrom
em-reinitialise-bedrock-client

Conversation

@ellenmuller
Copy link
Copy Markdown
Contributor

@ellenmuller ellenmuller commented Mar 11, 2026

What does this change?

Yesterday in the AWS office hour, we were told that the bedrock Client TCP connection is probably timing out after it is waiting for a response for bedrock for a long time. If true, this will exacerbate our backfill issues because it means that once we encounter the first hanging bedrock request, subsequent requests will always time out too because of the issue with the client.

Re-initialising the client each handler invocation won't fix the issues Bedrock server-side, but this might for now make our DLQ re-drive run more smoothly. It's worth a try!

See - https://aws.amazon.com/blogs/networking-and-content-delivery/implementing-long-running-tcp-connections-within-vpc-networking/

@ellenmuller ellenmuller added the maintenance Departmental tracking: maintenance work, not a fix or a feature label Mar 11, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 11, 2026

@ellenmuller ellenmuller marked this pull request as ready for review March 11, 2026 12:00
@ellenmuller ellenmuller requested a review from a team as a code owner March 11, 2026 12:00
@ellenmuller ellenmuller requested a review from joelochlann March 11, 2026 12:04
Copy link
Copy Markdown
Member

@joelochlann joelochlann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for actioning so quickly.

Only thing we might want to add is some timing logging around the client init, so we know how much it costs us in latency to re-init on every invocation. Could be useful for pressuring AWS to fix their dodgy SDK!

@gu-prout
Copy link
Copy Markdown

gu-prout Bot commented Mar 11, 2026

Seen on auth, usage, image-loader, metadata-editor, thrall, leases, cropper, collections, media-api, kahuna (merged by @ellenmuller 9 minutes and 46 seconds ago) Please check your changes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants