Skip to content

Function timeout after "functionTimeout": "00:10:00", minutes #44384

@georgekosmidis

Description

@georgekosmidis
  • Package Name: azure-servicebus
  • Package Version: 7.14.3
  • Operating System: Azure Functions v4 on Linux Premium (Containerized)]
  • Python Version: 3.11.13

Describe the bug

When I use the async Service Bus client (azure.servicebus.aio) inside a Python Azure Function that is triggered by Service Bus, the function invocation does not end cleanly. After exactly 5 minutes, the Functions host kills the invocation with a function timeout error, even though my handler has finished its work for the batch and I expect the invocation to complete.

This 5 minute timeout matches the configured functionTimeout in host.json.

If I switch the code to use the sync client from azure.servicebus with the same logic and the same load, the function exits normally within those 5 minutes and I do not see the timeout.

So the behavior is:

  • Async client inside Azure Functions
    function runs, then after about 5 minutes (the configured timeout) I see a log such as
    Function 'X' timeout after 00:05:00

  • Sync client inside Azure Functions
    function processes the same messages and exits, no timeout

This looks like some async resources or tasks from the async client staying alive in the worker, which keeps the invocation active until the host timeout hits.

To Reproduce

I am running a Service Bus trigger function in Python that reads messages from one queue and sends messages to another entity.

Steps

  1. Create a Python v4 Azure Function App with a Service Bus trigger.

  2. Install azure-servicebus and use the async client inside the function handler:

# service_bus_service.py

import asyncio
import azure.functions as func
from azure.servicebus.aio import ServiceBusClient
from azure.servicebus import ServiceBusMessage

SERVICE_BUS_CONNECTION_STR = "<connection string>"

class ServiceBusService:
	# ... other stuff
	async def send_many(messages):
		async with ServiceBusClient.from_connection_string(
		   conn_str=SERVICE_BUS_CONNECTION_STR
		) as client:
			async with client.get_queue_sender(queue_name="queue2") as sender:
				messages = [ServiceBusMessage(body) for body in bodies]
				await sender.send_messages(messages)
# azure_function.py 

import azure.functions as func
from typing import List

from myapp.service_bus_service import ServiceBusService
from myapp.settings import settings

postgres_worker_bp = func.Blueprint()
service_bus = ServiceBusService()

TYPE_ID = "type"

@postgres_worker_bp.service_bus_queue_trigger(
    arg_name="msgs",
    queue_name="queue1",
    connection="SERVICE_BUS_ENDPOINT",
    cardinality="many",
    max_batch_size=200,
)
async def postgres_proxy(msgs: List[func.ServiceBusMessage]) -> None:
    # For each incoming Service Bus message I build some outgoing messages
    bodies = [m.get_body().decode("utf8") for m in msgs]

    # This uses azure.servicebus.aio.ServiceBusClient internally
    # to send a batch of new Service Bus messages
    await service_bus.send_many(bodies)
  1. Configure the Service Bus trigger so that it receives messages at a steady rate
    for example around 200k messages per hour, with normal maxConcurrentCalls and prefetchCount settings.

  2. Let the function run under load for more than one hour.

Actual result

  • After about 5 minutes, the Functions host logs a timeout like
    Timeout value of 00:5:00 was exceeded by function: Functions.<name>
  • App Insights shows that the function timed out, even though the user code path finishes and there is no explicit infinite loop in my handler.

Control test with sync client

If I change the send_many implementation to use the sync client:

from azure.servicebus import ServiceBusClient, ServiceBusMessage

def send_many_sync(messages):
    with ServiceBusClient.from_connection_string(
        conn_str=SERVICE_BUS_CONNECTION_STR
    ) as client:
        sender = client.get_queue_sender(queue_name=OUTPUT_QUEUE_NAME)
        with sender:
			messages = [ServiceBusMessage(body) for body in bodies]
			sender.send_messages(messages)

and call that from main instead of the async version, the function runs under the same load without hitting the 5 minute timeout. Invocations complete and return as expected.

Expected behavior

  • Using the async azure.servicebus.aio.ServiceBusClient inside a Python Azure Function should not keep the invocation alive after user code completes.
  • The async version should behave the same as the sync version from the Functions host point of view.
  • Once my handler finishes (in this case after sending a batch of messages), the runtime should see the invocation as complete and should not hit the functionTimeout limit.

Screenshots

If needed I can provide:

  • Screenshot of the Application Insights trace that shows
    Timeout value of 00:05:00 was exceeded by function
  • Host logs that show the function work finishing before the timeout

Additional context

  • The function is a Service Bus triggered Python function that reads messages from one queue and pushes new messages to another queue or topic.
  • Under load I need to handle around 200k messages per hour. With the sync client the throughput is a bit lower but stable. With the async client throughput is better but the invocation hits the 59 minute timeout.
  • The timeout value is set in host.json as functionTimeout: "00:05:00". I understand that this is the host limit. The point here is that the async client seems to keep the invocation active even when there is no ongoing user code work, while the sync client does not.
  • It feels like some internal async task or open link from azure.servicebus.aio is not cleaned up in the Functions worker, so the runtime thinks the invocation is still running until the timeout.

I also read this in TROUBLESHOOTING:

Mixing synchronous and asynchronous Service Bus operations can cause issues such as the AutoLockRenewer hanging indefinitely because the event loop is blocked. Ensure that blocking calls are not made when receiving and processing messages asynchronously.

But unfortunately i cannot see any issue with mixing operations from my side.

If there is any recommended pattern for using azure.servicebus.aio safely inside Python Azure Functions, guidance would also help.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ClientThis issue points to a problem in the data-plane of the library.Service AttentionWorkflow: This issue is responsible by Azure service team.Service Buscustomer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions