Performance [P2]: Reuse sandbox registration transport on retries

[berndverst]

What is the issue?

Each retriable sandbox registration failure creates and closes a new gRPC transport, channel, interceptor, and synchronous token manager.

What is the impact?

Repeated transient failures cause avoidable connection setup and authentication work, extending recovery time and adding load during an outage or unstable network period.

Details about the issue including code reference

Relevant code:

durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/preview/sandboxes/worker.py

Lines 131 to 156 in 55d8e0b

	def _run_sandbox_registration_loop(self) -> None:
	retry_delay = 1.0
	while not self._sandbox_registration_stop.is_set():
	try:
	client = SandboxActivitiesGrpcTransport(
	host_address=self._sandbox_host_address,
	taskhub=self._sandbox_taskhub,
	token_credential=self._sandbox_token_credential,
	secure_channel=self._sandbox_secure_channel)
	try:
	client.connect_sandbox_activity_worker(self._registration_messages())
	retry_delay = 1.0
	finally:
	client.close()
	except Exception as ex:
	if self._sandbox_registration_stop.is_set():
	break
	if not _is_retriable_registration_failure(ex):
	self._sandbox_logger.error(
	"Sandbox activity worker registration failed permanently: %s", ex)
	self._sandbox_registration_stop.set()
	break
	self._sandbox_logger.warning("Sandbox activity worker registration failed: %s", ex)
	delay = random.uniform(0, retry_delay)
	self._sandbox_registration_stop.wait(delay)
	retry_delay = min(retry_delay * 2, 30.0)

durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/preview/sandboxes/transport.py

Lines 50 to 62 in 55d8e0b

	self._owns_channel = channel is None
	if channel is None:
	resolved_interceptors: list[shared.ClientInterceptor] = (
	list(interceptors) if interceptors is not None else []
	)
	resolved_interceptors.append(DTSDefaultClientInterceptorImpl(token_credential, taskhub))
	channel = shared.get_grpc_channel(
	host_address=host_address,
	secure_channel=secure_channel,
	interceptors=resolved_interceptors,
	channel_options=channel_options)
	self._channel = channel
	self._stub = cast(_SandboxActivitiesStub, stubs.SandboxActivitiesStub(channel))

durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/internal/durabletask_grpc_interceptor.py

Lines 44 to 50 in 55d8e0b

	self._token_manager = None
	if token_credential is not None:
	self._token_credential = token_credential
	self._token_manager = AccessTokenManager(token_credential=self._token_credential)
	access_token = self._token_manager.get_access_token()
	if access_token is not None:
	self._upsert_authorization_header(access_token.token)

SandboxWorker._run_sandbox_registration_loop() creates SandboxActivitiesGrpcTransport inside every retry iteration. The transport creates a new channel, stub, and DTSDefaultClientInterceptorImpl when no channel is supplied. The synchronous interceptor currently acquires a token during construction.

A potential or proposed solution

Create one transport for the registration loop and reuse it across retriable stream failures, closing it when the loop exits. Recreate it only after failures known to leave the channel unusable, while preserving the existing retry backoff, stop signal, and thread-join behavior.

Cold-start relevance

This affects the first sandbox registration attempt as well as retries. SandboxWorker first constructs its parent worker, then the registration loop creates a separate SandboxActivitiesGrpcTransport with another synchronous interceptor and token manager. Reusing a transport across retries does not remove that initial duplicate authentication setup; consider sharing compatible channel or authentication state with the parent worker when safe.

Relevant code:

durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/preview/sandboxes/worker.py

Lines 43 to 64 in 55d8e0b

	def __init__(self) -> None:
	resolved_host_address = _resolve_host_address()
	resolved_taskhub = _resolve_taskhub()
	resolved_secure_channel = _resolve_secure_channel(resolved_host_address)
	resolved_token_credential = _resolve_token_credential()
	resolved_max_concurrent_activities = _resolve_max_concurrent_activities()
	resolved_dts_sandbox_identifier = _resolve_dts_sandbox_identifier()
	resolved_sandbox_provider = _resolve_sandbox_provider()
	concurrency_options = ConcurrencyOptions(
	maximum_concurrent_activity_work_items=resolved_max_concurrent_activities)
	self._sandbox_host_address = resolved_host_address
	self._sandbox_secure_channel = resolved_secure_channel
	self._sandbox_token_credential = resolved_token_credential
	self._sandbox_logger = shared.get_logger("worker")
	super().__init__(
	host_address=resolved_host_address,
	taskhub=resolved_taskhub,
	token_credential=resolved_token_credential,
	secure_channel=resolved_secure_channel,
	concurrency_options=concurrency_options)

durabletask-python/durabletask-azuremanaged/durabletask/azuremanaged/preview/sandboxes/worker.py

Lines 131 to 144 in 55d8e0b

	def _run_sandbox_registration_loop(self) -> None:
	retry_delay = 1.0
	while not self._sandbox_registration_stop.is_set():
	try:
	client = SandboxActivitiesGrpcTransport(
	host_address=self._sandbox_host_address,
	taskhub=self._sandbox_taskhub,
	token_credential=self._sandbox_token_credential,
	secure_channel=self._sandbox_secure_channel)
	try:
	client.connect_sandbox_activity_worker(self._registration_messages())
	retry_delay = 1.0
	finally:
	client.close()