Possible regression introduced in 1.11.801 (2nd attempt)

[oleg-boulanov]

Describe the bug

In #3809, description states that "This is fully backwards compatible. No code changes required."

This may be not entirely correct. For example, after using Bedrock's InvokeModelWithBidirectionalStreamAsync(), Aws::ShutdownAPI(options) lock indefinitely.

Regression confirmed by building and using both 1.11.800 and 1.11.801 version with the same test code.

#include "pch.h"

// EZA-28656: standalone repro of Aws::ShutdownAPI() hanging forever after a Bedrock Runtime
// InvokeModelWithBidirectionalStream call, for reporting to aws/aws-sdk-cpp.
//
// This file is intentionally independent of every Eliza:: type - no SonicEngine, no AwsSdk
// wrapper, no InputEvent helpers. It uses only the AWS SDK for C++ (aws-cpp-sdk-core,
// aws-cpp-sdk-bedrock-runtime) plus the C++ standard library, so it can be lifted verbatim into
// a plain main() or a fresh aws-sdk-cpp sample project.
//
// Root cause (confirmed by live WinDbg inspection of the hung process, see EZA-28656): the
// generated Aws::BedrockRuntime::BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync
// installs a signing callback on the event-encoder stream that captures the stream's own
// shared_ptr by value - a self-reference cycle. Since aws-sdk-cpp PR #3809 ("use write data API
// in bidirectional streaming", first shipped in v1.11.801), that stream's backing buffer
// (HttpWriteDataStreamBuf) holds a live HTTP/2 Connection/ClientStream for as long as the cycle
// keeps it alive - so the cycle now pins a real connection forever instead of an inert buffer.
// Aws::ShutdownAPI() -> Aws::CleanupCrt() then blocks forever waiting for the CRT's native
// event-loop-group to report fully stopped, because that pinned connection never releases it.
//
// This repro deliberately does NOT work around the bug (unlike this repo's own SonicEngine,
// which calls streamInput.SetSigningCallback(nullptr) as a mitigation) - the whole point is to
// show the raw upstream behavior.
//
// Aws::InitAPI()/Aws::ShutdownAPI() must run on the SAME OS thread - Aws.h says so explicitly:
// "Please call this from the same thread from which InitAPI() has been called (use a dedicated
// thread if necessary)." So both calls run on one dedicated thread here (started before, and
// joined after, the Bedrock call), and that thread is given a bounded 5 seconds to finish before
// this test reports the outcome and returns - so this test is safe to run on its own, with no
// external timeout/process-kill needed, even though the bug it demonstrates is a genuine
// unbounded native hang.
//
// How to run (isolated, since it exercises real AWS SDK global init/shutdown state):
// Eliza.IVR.Sonic.ClientLib.Tests.exe --gtest_filter=Eza28656ShutdownHangRepro.ShutdownHangsAfterBidirectionalStream
// If it reproduces, this test FAILS with "Aws::ShutdownAPI() did not complete within 5000ms" -
// that failure IS the demonstration of the bug. Its dedicated thread is left running in the
// background (there's no way to safely abandon a thread that's stuck inside a native SDK call).
//
// Requires: valid AWS credentials in the default profile (~/.aws/credentials) with
// bedrock:InvokeModelWithBidirectionalStream permission for the model below, in the given region.

// This NuGet package builds aws-sdk-cpp/aws-crt-cpp as DLLs, not static libs - consumers must
// define these before including any AWS header (normally via that package's own
// eliza/aws-sdk-cpp-config.h, deliberately NOT included here to keep this file free of anything
// under an eliza/ path). Without AWS_NO_STATIC_IMPL, aws-c-common's "static inline" helper
// functions get compiled directly into this translation unit instead of imported from the DLL -
// and one of those (aws/common/math.h) picks its x64-only MSVC intrinics header
// (math.msvc_x64.inl, using _umul128/_addcarry_u64/_BitScanReverse64) even for this x86 build, an
// unrelated packaging bug that only surfaces when the code is compiled inline like this.
#define AWS_NO_STATIC_IMPL
#define USE_IMPORT_EXPORT
#define USE_WINDOWS_DLL_SEMANTICS

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/core/utils/json/JsonSerializer.h>

#include <aws/bedrock-runtime/BedrockRuntimeClient.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamRequest.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamHandler.h>
#include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamInput.h>
#include <aws/bedrock-runtime/model/BidirectionalInputPayloadPart.h>

#include
#include
#include <condition_variable>
#include
#include
#include
#include
#include

namespace {

// A random UUID isn't needed here - Nova Sonic just needs promptName/contentName to be unique
// within this one session. (Aws::Utils::UUID is deliberately avoided: it pulls in an aws-c-common
// header, math.msvc_x64.inl, that turned out to be broken for this project's x86 build - a
// separate, unrelated packaging issue not worth fighting for a minimal repro.)
Aws::String MakeId(const char* prefix)
{
static std::atomic counter{ 0 };
return Aws::String(prefix) + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count())
+ "-" + std::to_string(++counter);
}

// Owns the one dedicated thread that Aws::InitAPI()/Aws::ShutdownAPI() both run on.
struct ApiThread {
std::thread thread;
std::promise shutdownRequested;
};

std::unique_ptr StartApiThread(Aws::SDKOptions options)
{
auto state = std::make_unique();
std::promise initDone;
std::future initDoneFuture = initDone.get_future();
std::shared_future shutdownRequestedFuture = state->shutdownRequested.get_future().share();
state->thread = std::thread(options, initDone = std::move(initDone), shutdownRequestedFuture mutable {
Aws::InitAPI(options);
initDone.set_value();
shutdownRequestedFuture.wait();
Aws::ShutdownAPI(options);
});
initDoneFuture.wait();
return state;
}

// Signals the dedicated thread to proceed to Aws::ShutdownAPI(), then waits up to timeout for
// it to finish. The actual join happens on a THIRD thread - joining a std::thread has no
// thread-affinity requirement, unlike the AWS calls it's waiting on - so that this function itself
// can safely time out without violating the same-thread rule it exists to respect. Returns true if
// ShutdownAPI() completed in time; false if it's still stuck (its thread is then abandoned,
// running in the background - there's no safe way to cancel a thread blocked inside a native SDK
// call).
bool StopApiThreadWithTimeout(std::unique_ptr state, std::chrono::milliseconds timeout)
{
state->shutdownRequested.set_value();
std::thread apiThread = std::move(state->thread);
auto done = std::make_shared<std::promise>();
std::future doneFuture = done->get_future();
std::thread(apiThread = std::move(apiThread), done mutable {
apiThread.join();
done->set_value();
}).detach();
return doneFuture.wait_for(timeout) == std::future_status::ready;
}

// Builds one Nova Sonic input event as compact JSON: {"event":{: }}.
Aws::String WrapEvent(const Aws::String& eventName, Aws::Utils::Json::JsonValue&& body)
{
Aws::Utils::Json::JsonValue event;
event.WithObject(eventName, std::move(body));
Aws::Utils::Json::JsonValue wrapper;
wrapper.WithObject("event", std::move(event));
return wrapper.View().WriteCompact();
}

void SendEvent(
Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput,
const Aws::String& json)
{
std::cout << ">> " << json << std::endl;
Aws::BedrockRuntime::Model::BidirectionalInputPayloadPart part;
part.SetBytes(Aws::Utils::CryptoBuffer(reinterpret_cast<const unsigned char*>(json.c_str()), json.size()));
streamInput.WriteBidirectionalInputPayloadPart(part);
}

} // namespace

TEST(Eza28656ShutdownHangRepro, ShutdownHangsAfterBidirectionalStream)
{
Aws::SDKOptions options;
auto apiThread = StartApiThread(options);

{
	Aws::Client::ClientConfigurationInitValues initValues{ /*shouldDisableIMDS:*/ true };
	Aws::Client::ClientConfiguration clientConfiguration(initValues);
	clientConfiguration.region = "us-east-1";
	clientConfiguration.requestTimeoutMs = 30000;

	Aws::Auth::ProfileConfigFileAWSCredentialsProvider credentialsProvider("default");
	Aws::Auth::AWSCredentials credentials = credentialsProvider.GetAWSCredentials();
	ASSERT_FALSE(credentials.IsEmpty()) << "No AWS credentials found in the default profile";

	auto pClient = Aws::MakeShared<Aws::BedrockRuntime::BedrockRuntimeClient>(
		"Eza28656Repro", credentials, clientConfiguration);

	const Aws::String promptName = MakeId("prompt-");
	const Aws::String contentName = MakeId("content-");

	std::mutex doneMutex;
	std::condition_variable doneCv;
	bool outcomeReceived = false;

	Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamHandler streamHandler;
	streamHandler.SetInitialResponseCallbackEx(
		[](const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInitialResponse& response,
			Aws::Utils::Event::InitialResponseType) {
				std::cout << "<< InitialResponse: " << response.Jsonize().View().WriteCompact() << std::endl;
		});
	streamHandler.SetOnErrorCallback(
		[](const Aws::Client::AWSError<Aws::BedrockRuntime::BedrockRuntimeErrors>& error) {
			std::cout << "<< StreamError: " << error.GetMessage() << std::endl;
		});
	streamHandler.SetBidirectionalOutputPayloadPartCallback(
		[](const Aws::BedrockRuntime::Model::BidirectionalOutputPayloadPart& part) {
			const auto& bytes = part.GetBytes();
			std::cout << "<< " << Aws::String(reinterpret_cast<const char*>(bytes.GetUnderlyingData()), bytes.GetLength()) << std::endl;
		});

	Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest streamingRequest;
	streamingRequest.SetEventStreamHandler(streamHandler);
	streamingRequest.SetModelId("amazon.nova-2-sonic-v1:0");

	pClient->InvokeModelWithBidirectionalStreamAsync(
		streamingRequest,
		[promptName, contentName](Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput) {
			// Minimal session: start, one system-prompt turn, then close - no audio content
			// needed. The bug is structural (a shared_ptr cycle created as soon as the stream
			// exists), not dependent on how much of a real conversation happens.
			Aws::Utils::Json::JsonValue inferenceConfiguration;
			inferenceConfiguration.WithInteger("maxTokens", 1024).WithDouble("topP", 0.9).WithDouble("temperature", 0.7);
			Aws::Utils::Json::JsonValue sessionStart;
			sessionStart.WithObject("inferenceConfiguration", inferenceConfiguration);
			SendEvent(streamInput, WrapEvent("sessionStart", std::move(sessionStart)));

			Aws::Utils::Json::JsonValue textOutputConfiguration;
			textOutputConfiguration.WithString("mediaType", "text/plain");
			Aws::Utils::Json::JsonValue promptStart;
			promptStart.WithString("promptName", promptName).WithObject("textOutputConfiguration", textOutputConfiguration);
			SendEvent(streamInput, WrapEvent("promptStart", std::move(promptStart)));

			Aws::Utils::Json::JsonValue textInputConfiguration;
			textInputConfiguration.WithString("mediaType", "text/plain");
			Aws::Utils::Json::JsonValue contentStart;
			contentStart.WithString("promptName", promptName)
				.WithString("contentName", contentName)
				.WithString("type", "TEXT")
				.WithString("role", "SYSTEM")
				.WithObject("textInputConfiguration", textInputConfiguration);
			SendEvent(streamInput, WrapEvent("contentStart", std::move(contentStart)));

			Aws::Utils::Json::JsonValue textInput;
			textInput.WithString("promptName", promptName).WithString("contentName", contentName).WithString("content", "Keep responses short.");
			SendEvent(streamInput, WrapEvent("textInput", std::move(textInput)));

			Aws::Utils::Json::JsonValue contentEnd;
			contentEnd.WithString("promptName", promptName).WithString("contentName", contentName);
			SendEvent(streamInput, WrapEvent("contentEnd", std::move(contentEnd)));

			Aws::Utils::Json::JsonValue promptEnd;
			promptEnd.WithString("promptName", promptName);
			SendEvent(streamInput, WrapEvent("promptEnd", std::move(promptEnd)));

			SendEvent(streamInput, WrapEvent("sessionEnd", Aws::Utils::Json::JsonValue()));

			streamInput.Close();
			// Deliberately NOT calling streamInput.SetSigningCallback(nullptr) here - that is
			// this repo's own workaround (see SonicEngine::OnStreamReady), and the whole point
			// of this file is to reproduce the raw, un-worked-around upstream bug.
			std::cout << "Stream closed." << std::endl;
		},
		[&doneMutex, &doneCv, &outcomeReceived](
			const Aws::BedrockRuntime::BedrockRuntimeClient*,
			const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest&,
			const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamOutcome& outcome,
			const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) {
				std::cout << "Final outcome: " << (outcome.IsSuccess() ? Aws::String("SUCCESS") : outcome.GetError().GetMessage()) << std::endl;
				{
					std::lock_guard<std::mutex> lock(doneMutex);
					outcomeReceived = true;
				}
				doneCv.notify_all();
		},
		nullptr);

	{
		std::unique_lock<std::mutex> lock(doneMutex);
		ASSERT_TRUE(doneCv.wait_for(lock, std::chrono::seconds(30), [&] { return outcomeReceived; }))
			<< "InvokeModelWithBidirectionalStreamAsync did not complete within 30s - this is a "
			<< "different failure than EZA-28656 (which happens AFTER a successful call)";
	}

	std::cout << "pClient.use_count() before reset: " << pClient.use_count() << std::endl;
	pClient.reset();
}	// options/credentials/etc. go out of scope here; only apiThread itself outlives this block

// This is the call under test. If EZA-28656 reproduces, Aws::CleanupCrt() blocks forever
// inside Aws::ShutdownAPI(), waiting for the CRT event-loop group to report fully stopped -
// but this wait is bounded, so this test reports that outcome instead of hanging the process.
std::cout << "Requesting Aws::ShutdownAPI()..." << std::endl;
auto t0 = std::chrono::steady_clock::now();
bool completed = StopApiThreadWithTimeout(std::move(apiThread), std::chrono::seconds(5));
auto elapsedMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - t0).count();
if (completed) {
	std::cout << "Aws::ShutdownAPI() returned after " << elapsedMs << "ms." << std::endl;
}
else {
	std::cout << "Aws::ShutdownAPI() did NOT complete within 5000ms - its dedicated thread is "
		<< "left running in the background." << std::endl;
}
EXPECT_TRUE(completed) << "Aws::ShutdownAPI() did not complete within 5000ms - EZA-28656 reproduced";

}

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Aws::ShutdownAPI() exits cleanly after limited amount of time

Current Behavior

Aws::ShutdownAPI() hangs

Reproduction Steps

Build AWS SDK:
"C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" ...... -G "Visual Studio 18 2026" -A Win32 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX="........\install" -DBUILD_ONLY="bedrock-runtime" -DTARGET_ARCH=Windows -DBUILD_SHARED_LIBS=ON -DUSE_OPENSSL=OFF -DUSE_CRT_HTTP_CLIENT=ON 

Build and run attached unit test using 1.11.800 and 1.11.801 ASW SDK versions

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

= 1.11.801

Compiler and Version used

Visual Studio 2026

Operating System and version

Windows 11

[sbiscigl]

AWS CPP SDK version used
= 1.11.801

from the last ticket:

actually we pushed a related fix in 1.11.832, so i would also suggest testing against main as the issue may already be resolved.

also you should see a large improvement on CPU usage after picking this up

did you try picking up a version post that patch and seeing if that resolves your issue? that specifically dealt with internal issue titled

AudioStream::WriteAudioEvent() and AudioStream::Close() block indefinitely (26+ minutes) when network drops during active Transcribe Streaming session

which sounds very similar to your case, sepcifically in you RCA

Aws::CleanupCrt() then blocks forever waiting for the CRT's native
event-loop-group to report fully stopped, because that pinned connection never releases it.

so i would suggest seeing if the issue is fixed on main before further debugging

[oleg-boulanov]

did you try picking up a version post that patch and seeing if that resolves your issue? Yes, tried the latest .882, also hangs

…

________________________________ From: Sam Bisciglia ***@***.***> Sent: Thursday, September 3, 2026 9:58 AM To: aws/aws-sdk-cpp ***@***.***> Cc: Boulanov, Oleg ***@***.***>; Author ***@***.***> Subject: [EXTERNAL] Re: [aws/aws-sdk-cpp] Possible regression introduced in 1.11.801 (2nd attempt) (Issue #3917) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. [https://outlook.office.com/mail/AAMkADRhMmU1MTI3LTYxYzQtNGNmNS04M2JkLTQ2NmM1MTUwMzRjMQAuAAAAAAAU0DbbCt0oQq3SL1Q7E%2F1BAQDYBzv4K3DTT7r18TRttj0qAAAM%2F6C4AAA%3D/id/AAkALgAAAAAAHYQDEapmEc2byACqAC%2FEWg0A2Ac7%2BCtw00%2B69fE0bbY9KgAE9ilPLQAA?nativeVersion=1.2026.818.100]sbiscigl left a comment (aws/aws-sdk-cpp#3917)<https://urldefense.com/v3/__https://github.com/aws/aws-sdk-cpp/issues/3917*issuecomment-5526898498__;Iw!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24ATt-0VaQ$> AWS CPP SDK version used = 1.11.801 from the last ticket<https://urldefense.com/v3/__https://github.com/aws/aws-sdk-cpp/issues/3911*issuecomment-5500176980__;Iw!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24AKo8BLOQ$>: actually we pushed a related fix in 1.11.832<https://urldefense.com/v3/__https://github.com/aws/aws-sdk-cpp/commit/683380e0a6723bf4d7e9209a569c572505bb6aa4__;!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24BvJq91NA$>, so i would also suggest testing against main as the issue may already be resolved. also you should see a large improvement on CPU usage after picking this up did you try picking up a version post that patch and seeing if that resolves your issue? that specifically dealt with internal issue titled AudioStream::WriteAudioEvent() and AudioStream::Close() block indefinitely (26+ minutes) when network drops during active Transcribe Streaming session which sounds very similar to your case, sepcifically in you RCA Aws::CleanupCrt() then blocks forever waiting for the CRT's native event-loop-group to report fully stopped, because that pinned connection never releases it. so i would suggest seeing if the issue is fixed on main before further debugging — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/aws/aws-sdk-cpp/issues/3917?email_source=notifications&email_token=AXU6KYHYORF2WN3XA5FVTLT5NF2HHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJSGY4DSOBUHE4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L*issuecomment-5526898498__;Iw!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24DIucvzYg$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AXU6KYAP5JZE7PBV2E6GHF35NF2HHAVCNFSNUABEKJSXA33TNF2G64TZHMZTKNBUGA2DANB3JFZXG5LFHM2TGMZWGQZTCOJSGCQXMAQ__;!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24DihltXcA$>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https://github.com/notifications/mobile/ios/AXU6KYAU3YHF6Y3PAF2N3S35NF2HHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJSGY4DSOBUHE4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG__;!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24CWKS_OZQ$> and Android<https://urldefense.com/v3/__https://github.com/notifications/mobile/android/AXU6KYFUTOWLBSFO4YT5SYT5NF2HHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJSGY4DSOBUHE4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA__;!!OnbYIJ6QcFHw!MD1FRlV50q25c2CAhvv4I4L5_5WN9-pW8OpgcDyQ6I9-r6Y48LmPvwsb8c8rFNw-oH4droGKfZjTDeBrsEsv24BZD9iLkA$>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[oleg-boulanov]

Nice, it looks like replying via email works! I discovered the issue while using a later build, 1.11.824, and started digging with Claude Code, which was able to narrow the regression down to the change made in 1.11.801: The claim is that PR #3809<#3809> ("use write data API in bidirectional streaming"), shipped in tag 1.11.801, causes a shared_ptr self-capture cycle in the generated BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync. I pulled the actual source at tag 1.11.881 (the latest release, published 2026-08-28) from generated/src/aws-cpp-sdk-bedrock-runtime/source/BedrockRuntimeClient.cpp and it still reads: eventEncoderStream->SetSigningCallback([this, ctx, eventEncoderStream](Message&, String&) -> bool { ... }); eventEncoderStream (a shared_ptr to itself) is captured by value and stored via EventEncoderStream::SetSigningCallback → m_encoder.SetSigningCallback(...), a member of the same object. That's a self-reference cycle, exactly as described. EventEncoderStream.h confirms the storage path unchanged. Fixed in the latest available version? No. I checked commit history since the ticket's last-checked date (2026-06-18) on all three relevant files: * BedrockRuntimeClient.cpp — one unrelated commit (allocation-strategy enums elsewhere in the same file). * SmithyJsonServiceEventStreamOperationsSource.vm (the code-gen template) — zero commits. * EventEncoderStream.h — zero commits. I also searched upstream issues/PRs for anything referencing this hang, self-reference cycle, or bidirectional-stream shutdown — nothing found. No one has filed the upstream issue yet either (the ticket's own "next steps" item). Three gaps in upstream test coverage explain why this regression shipped and stayed unnoticed: 1. No test at all for InvokeModelWithBidirectionalStreamAsync. The only Bedrock-runtime integration tests (tests/aws-cpp-sdk-bedrock-runtime-integration-tests/IntegrationTests.cpp) cover ConverseStream (server→client only) and plain InvokeModel. The bidirectional-streaming API — the one with the self-capture bug — has zero test coverage. 2. EventEncoderStream/SetSigningCallback (the write/encoder side) isn't tested. tests/aws-cpp-sdk-core-tests/utils/event/EventStreamTest.cpp only exercises EventDecoderStream — parsing/decoding incoming events (payload counts, headers, XML errors). Nothing touches the encoder side or the signing-callback path at all. 3. No lifecycle/leak/shutdown-hang test anywhere in the chain. Nothing checks shared_ptr ownership cycles, connection-object lifetime after stream completion, or bounds Aws::ShutdownAPI(). aws-crt-cpp has bootstrap-shutdown tests for other clients (ChannelBootstrapTest.cpp, Mqtt5ClientTest.cpp), but nothing exercises the generated Bedrock write-data-buffer (HttpWriteDataStreamBuf) path that ends up pinning the connection. So this is a genuine blind spot, not a known-and-accepted risk — no regression test would have caught PR #3809 introducing the leak, and none exists today to catch it if AWS ever tries to fix it. Reinforces the earlier finding: filing the upstream issue (still not done) should probably also ask for a bidirectional-stream teardown/leak test, since there's nothing there to regress against. Tried the latest (1.11.882) and the test still hangs. Rolling back to 1.11.801 fixes it. Please find the unit test code attached. #include "pch.h" // EZA-28656: standalone repro of Aws::ShutdownAPI() hanging forever after a Bedrock Runtime // InvokeModelWithBidirectionalStream call, for reporting to aws/aws-sdk-cpp. // // This file is intentionally independent of every Eliza:: type - no SonicEngine, no AwsSdk // wrapper, no InputEvent helpers. It uses only the AWS SDK for C++ (aws-cpp-sdk-core, // aws-cpp-sdk-bedrock-runtime) plus the C++ standard library, so it can be lifted verbatim into // a plain main() or a fresh aws-sdk-cpp sample project. // // Root cause (confirmed by live WinDbg inspection of the hung process, see EZA-28656): the // generated Aws::BedrockRuntime::BedrockRuntimeClient::InvokeModelWithBidirectionalStreamAsync // installs a signing callback on the event-encoder stream that captures the stream's own // shared_ptr by value - a self-reference cycle. Since aws-sdk-cpp PR #3809 ("use write data API // in bidirectional streaming", first shipped in v1.11.801), that stream's backing buffer // (HttpWriteDataStreamBuf) holds a live HTTP/2 Connection/ClientStream for as long as the cycle // keeps it alive - so the cycle now pins a real connection forever instead of an inert buffer. // Aws::ShutdownAPI() -> Aws::CleanupCrt() then blocks forever waiting for the CRT's native // event-loop-group to report fully stopped, because that pinned connection never releases it. // // This repro deliberately does NOT work around the bug (unlike this repo's own SonicEngine, // which calls streamInput.SetSigningCallback(nullptr) as a mitigation) - the whole point is to // show the raw upstream behavior. // // Aws::InitAPI()/Aws::ShutdownAPI() must run on the SAME OS thread - Aws.h says so explicitly: // "Please call this from the same thread from which InitAPI() has been called (use a dedicated // thread if necessary)." So both calls run on one dedicated thread here (started before, and // joined after, the Bedrock call), and that thread is given a bounded 5 seconds to finish before // this test reports the outcome and returns - so this test is safe to run on its own, with no // external timeout/process-kill needed, even though the bug it demonstrates is a genuine // unbounded native hang. // // How to run (isolated, since it exercises real AWS SDK global init/shutdown state): // Eliza.IVR.Sonic.ClientLib.Tests.exe --gtest_filter=Eza28656ShutdownHangRepro.ShutdownHangsAfterBidirectionalStream // If it reproduces, this test FAILS with "Aws::ShutdownAPI() did not complete within 5000ms" - // that failure IS the demonstration of the bug. Its dedicated thread is left running in the // background (there's no way to safely abandon a thread that's stuck inside a native SDK call). // // Requires: valid AWS credentials in the default profile (~/.aws/credentials) with // bedrock:InvokeModelWithBidirectionalStream permission for the model below, in the given region. // This NuGet package builds aws-sdk-cpp/aws-crt-cpp as DLLs, not static libs - consumers must // define these before including any AWS header (normally via that package's own // eliza/aws-sdk-cpp-config.h, deliberately NOT included here to keep this file free of anything // under an eliza/ path). Without AWS_NO_STATIC_IMPL, aws-c-common's "static inline" helper // functions get compiled directly into this translation unit instead of imported from the DLL - // and one of those (aws/common/math.h) picks its x64-only MSVC intrinics header // (math.msvc_x64.inl, using _umul128/_addcarry_u64/_BitScanReverse64) even for this x86 build, an // unrelated packaging bug that only surfaces when the code is compiled inline like this. #define AWS_NO_STATIC_IMPL #define USE_IMPORT_EXPORT #define USE_WINDOWS_DLL_SEMANTICS #include <aws/core/Aws.h> #include <aws/core/auth/AWSCredentialsProvider.h> #include <aws/core/utils/json/JsonSerializer.h> #include <aws/bedrock-runtime/BedrockRuntimeClient.h> #include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamRequest.h> #include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamHandler.h> #include <aws/bedrock-runtime/model/InvokeModelWithBidirectionalStreamInput.h> #include <aws/bedrock-runtime/model/BidirectionalInputPayloadPart.h> #include <atomic> #include <chrono> #include <condition_variable> #include <future> #include <iostream> #include <memory> #include <mutex> #include <thread> namespace { // A random UUID isn't needed here - Nova Sonic just needs promptName/contentName to be unique // within this one session. (Aws::Utils::UUID is deliberately avoided: it pulls in an aws-c-common // header, math.msvc_x64.inl, that turned out to be broken for this project's x86 build - a // separate, unrelated packaging issue not worth fighting for a minimal repro.) Aws::String MakeId(const char* prefix) { static std::atomic<int> counter{ 0 }; return Aws::String(prefix) + std::to_string(std::chrono::steady_clock::now().time_since_epoch().count()) + "-" + std::to_string(++counter); } // Owns the one dedicated thread that Aws::InitAPI()/Aws::ShutdownAPI() both run on. struct ApiThread { std::thread thread; std::promise<void> shutdownRequested; }; std::unique_ptr<ApiThread> StartApiThread(Aws::SDKOptions options) { auto state = std::make_unique<ApiThread>(); std::promise<void> initDone; std::future<void> initDoneFuture = initDone.get_future(); std::shared_future<void> shutdownRequestedFuture = state->shutdownRequested.get_future().share(); state->thread = std::thread([options, initDone = std::move(initDone), shutdownRequestedFuture]() mutable { Aws::InitAPI(options); initDone.set_value(); shutdownRequestedFuture.wait(); Aws::ShutdownAPI(options); }); initDoneFuture.wait(); return state; } // Signals the dedicated thread to proceed to Aws::ShutdownAPI(), then waits up to `timeout` for // it to finish. The actual join happens on a THIRD thread - joining a std::thread has no // thread-affinity requirement, unlike the AWS calls it's waiting on - so that this function itself // can safely time out without violating the same-thread rule it exists to respect. Returns true if // ShutdownAPI() completed in time; false if it's still stuck (its thread is then abandoned, // running in the background - there's no safe way to cancel a thread blocked inside a native SDK // call). bool StopApiThreadWithTimeout(std::unique_ptr<ApiThread> state, std::chrono::milliseconds timeout) { state->shutdownRequested.set_value(); std::thread apiThread = std::move(state->thread); auto done = std::make_shared<std::promise<void>>(); std::future<void> doneFuture = done->get_future(); std::thread([apiThread = std::move(apiThread), done]() mutable { apiThread.join(); done->set_value(); }).detach(); return doneFuture.wait_for(timeout) == std::future_status::ready; } // Builds one Nova Sonic input event as compact JSON: {"event":{<eventName>: <body>}}. Aws::String WrapEvent(const Aws::String& eventName, Aws::Utils::Json::JsonValue&& body) { Aws::Utils::Json::JsonValue event; event.WithObject(eventName, std::move(body)); Aws::Utils::Json::JsonValue wrapper; wrapper.WithObject("event", std::move(event)); return wrapper.View().WriteCompact(); } void SendEvent( Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput, const Aws::String& json) { std::cout << ">> " << json << std::endl; Aws::BedrockRuntime::Model::BidirectionalInputPayloadPart part; part.SetBytes(Aws::Utils::CryptoBuffer(reinterpret_cast<const unsigned char*>(json.c_str()), json.size())); streamInput.WriteBidirectionalInputPayloadPart(part); } } // namespace TEST(Eza28656ShutdownHangRepro, ShutdownHangsAfterBidirectionalStream) { Aws::SDKOptions options; auto apiThread = StartApiThread(options); { Aws::Client::ClientConfigurationInitValues initValues{ /*shouldDisableIMDS:*/ true }; Aws::Client::ClientConfiguration clientConfiguration(initValues); clientConfiguration.region = "us-east-1"; clientConfiguration.requestTimeoutMs = 30000; Aws::Auth::ProfileConfigFileAWSCredentialsProvider credentialsProvider("default"); Aws::Auth::AWSCredentials credentials = credentialsProvider.GetAWSCredentials(); ASSERT_FALSE(credentials.IsEmpty()) << "No AWS credentials found in the default profile"; auto pClient = Aws::MakeShared<Aws::BedrockRuntime::BedrockRuntimeClient>( "Eza28656Repro", credentials, clientConfiguration); const Aws::String promptName = MakeId("prompt-"); const Aws::String contentName = MakeId("content-"); std::mutex doneMutex; std::condition_variable doneCv; bool outcomeReceived = false; Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamHandler streamHandler; streamHandler.SetInitialResponseCallbackEx( [](const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInitialResponse& response, Aws::Utils::Event::InitialResponseType) { std::cout << "<< InitialResponse: " << response.Jsonize().View().WriteCompact() << std::endl; }); streamHandler.SetOnErrorCallback( [](const Aws::Client::AWSError<Aws::BedrockRuntime::BedrockRuntimeErrors>& error) { std::cout << "<< StreamError: " << error.GetMessage() << std::endl; }); streamHandler.SetBidirectionalOutputPayloadPartCallback( [](const Aws::BedrockRuntime::Model::BidirectionalOutputPayloadPart& part) { const auto& bytes = part.GetBytes(); std::cout << "<< " << Aws::String(reinterpret_cast<const char*>(bytes.GetUnderlyingData()), bytes.GetLength()) << std::endl; }); Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest streamingRequest; streamingRequest.SetEventStreamHandler(streamHandler); streamingRequest.SetModelId("amazon.nova-2-sonic-v1:0"); pClient->InvokeModelWithBidirectionalStreamAsync( streamingRequest, [promptName, contentName](Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamInput& streamInput) { // Minimal session: start, one system-prompt turn, then close - no audio content // needed. The bug is structural (a shared_ptr cycle created as soon as the stream // exists), not dependent on how much of a real conversation happens. Aws::Utils::Json::JsonValue inferenceConfiguration; inferenceConfiguration.WithInteger("maxTokens", 1024).WithDouble("topP", 0.9).WithDouble("temperature", 0.7); Aws::Utils::Json::JsonValue sessionStart; sessionStart.WithObject("inferenceConfiguration", inferenceConfiguration); SendEvent(streamInput, WrapEvent("sessionStart", std::move(sessionStart))); Aws::Utils::Json::JsonValue textOutputConfiguration; textOutputConfiguration.WithString("mediaType", "text/plain"); Aws::Utils::Json::JsonValue promptStart; promptStart.WithString("promptName", promptName).WithObject("textOutputConfiguration", textOutputConfiguration); SendEvent(streamInput, WrapEvent("promptStart", std::move(promptStart))); Aws::Utils::Json::JsonValue textInputConfiguration; textInputConfiguration.WithString("mediaType", "text/plain"); Aws::Utils::Json::JsonValue contentStart; contentStart.WithString("promptName", promptName) .WithString("contentName", contentName) .WithString("type", "TEXT") .WithString("role", "SYSTEM") .WithObject("textInputConfiguration", textInputConfiguration); SendEvent(streamInput, WrapEvent("contentStart", std::move(contentStart))); Aws::Utils::Json::JsonValue textInput; textInput.WithString("promptName", promptName).WithString("contentName", contentName).WithString("content", "Keep responses short."); SendEvent(streamInput, WrapEvent("textInput", std::move(textInput))); Aws::Utils::Json::JsonValue contentEnd; contentEnd.WithString("promptName", promptName).WithString("contentName", contentName); SendEvent(streamInput, WrapEvent("contentEnd", std::move(contentEnd))); Aws::Utils::Json::JsonValue promptEnd; promptEnd.WithString("promptName", promptName); SendEvent(streamInput, WrapEvent("promptEnd", std::move(promptEnd))); SendEvent(streamInput, WrapEvent("sessionEnd", Aws::Utils::Json::JsonValue())); streamInput.Close(); // Deliberately NOT calling streamInput.SetSigningCallback(nullptr) here - that is // this repo's own workaround (see SonicEngine::OnStreamReady), and the whole point // of this file is to reproduce the raw, un-worked-around upstream bug. std::cout << "Stream closed." << std::endl; }, [&doneMutex, &doneCv, &outcomeReceived]( const Aws::BedrockRuntime::BedrockRuntimeClient*, const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamRequest&, const Aws::BedrockRuntime::Model::InvokeModelWithBidirectionalStreamOutcome& outcome, const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) { std::cout << "Final outcome: " << (outcome.IsSuccess() ? Aws::String("SUCCESS") : outcome.GetError().GetMessage()) << std::endl; { std::lock_guard<std::mutex> lock(doneMutex); outcomeReceived = true; } doneCv.notify_all(); }, nullptr); { std::unique_lock<std::mutex> lock(doneMutex); ASSERT_TRUE(doneCv.wait_for(lock, std::chrono::seconds(30), [&] { return outcomeReceived; })) << "InvokeModelWithBidirectionalStreamAsync did not complete within 30s - this is a " << "different failure than EZA-28656 (which happens AFTER a successful call)"; } std::cout << "pClient.use_count() before reset: " << pClient.use_count() << std::endl; pClient.reset(); } // options/credentials/etc. go out of scope here; only apiThread itself outlives this block // This is the call under test. If EZA-28656 reproduces, Aws::CleanupCrt() blocks forever // inside Aws::ShutdownAPI(), waiting for the CRT event-loop group to report fully stopped - // but this wait is bounded, so this test reports that outcome instead of hanging the process. std::cout << "Requesting Aws::ShutdownAPI()..." << std::endl; auto t0 = std::chrono::steady_clock::now(); bool completed = StopApiThreadWithTimeout(std::move(apiThread), std::chrono::seconds(5)); auto elapsedMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - t0).count(); if (completed) { std::cout << "Aws::ShutdownAPI() returned after " << elapsedMs << "ms." << std::endl; } else { std::cout << "Aws::ShutdownAPI() did NOT complete within 5000ms - its dedicated thread is " << "left running in the background." << std::endl; } EXPECT_TRUE(completed) << "Aws::ShutdownAPI() did not complete within 5000ms - EZA-28656 reproduced"; }

[sbiscigl]

small update, we were able to reproduce and are working on a fix we have that fix staged on a branch, will let you know when we get it out eddb695