Description
Delayed entity signals intermittently never reach the entity in the Azure Functions Durable E2E suite. This affects both client-scheduled signals and signals scheduled from an orchestration.
The failure reproduced independently in both strict-typing matrices in GitHub Actions run 30422946249:
test_client_delayed_signal_is_deferredtest_orchestration_delayed_signal_is_deferred
The suite used Durable Functions extension 3.14.0, Azurite 3.36.0, and maxQueuePollingInterval set to one second.
Observed behavior
Host logs show the scheduled EventRaised control message is fetched slightly before the timestamp encoded in its op@<timestamp> event name. DurableTask.Core reschedules the same logical request to the entity. Because the remaining sub-second delay is serialized to Azurite as visibilitytimeout=0, several newly-created queue messages are processed in rapid succession.
For one example scheduled for 04:43:59.5057260Z, the request was first fetched at 04:43:59.344Z and rescheduled repeatedly between 04:43:59.348Z and 04:43:59.408Z. Each iteration successfully enqueued the replacement and checkpointed entity history before deleting its triggering message. The final iteration checkpointed and deleted the trigger without invoking the entity operation; entity state remained absent for the full 30-second polling timeout.
This is not control-queue poison-message exhaustion. DurableTask.AzureStorage has no maximum dequeue count that deletes or abandons a control message. Abandoned messages back off to a maximum 10-minute visibility delay and continue retrying; reaching that delay only emits a poison-message log. In this failure, each early-delivery iteration creates a new queue message, so its dequeue count starts over.
Expected behavior
A scheduled entity operation should remain pending until its due time and then execute exactly once, even when the storage emulator exposes the queue message slightly early.
Possible direction
Investigate the final due-time transition in TaskEntityDispatcher.DetermineWork and the entity scheduler's duplicate/message-sorter state. Separately, round a positive sub-second remaining duration up to a nonzero queue visibility delay, or otherwise prevent the immediate early-delivery loop under Azurite.
Reproduction
Run:
nox -s functions_e2e -- tests/azure-functions-durable/e2e/test_dtask_entities_advanced_e2e.py
The issue is timing-dependent; the full E2E suite increases reproduction frequency.
Description
Delayed entity signals intermittently never reach the entity in the Azure Functions Durable E2E suite. This affects both client-scheduled signals and signals scheduled from an orchestration.
The failure reproduced independently in both strict-typing matrices in GitHub Actions run
30422946249:test_client_delayed_signal_is_deferredtest_orchestration_delayed_signal_is_deferredThe suite used Durable Functions extension
3.14.0, Azurite3.36.0, andmaxQueuePollingIntervalset to one second.Observed behavior
Host logs show the scheduled
EventRaisedcontrol message is fetched slightly before the timestamp encoded in itsop@<timestamp>event name. DurableTask.Core reschedules the same logical request to the entity. Because the remaining sub-second delay is serialized to Azurite asvisibilitytimeout=0, several newly-created queue messages are processed in rapid succession.For one example scheduled for
04:43:59.5057260Z, the request was first fetched at04:43:59.344Zand rescheduled repeatedly between04:43:59.348Zand04:43:59.408Z. Each iteration successfully enqueued the replacement and checkpointed entity history before deleting its triggering message. The final iteration checkpointed and deleted the trigger without invoking the entity operation; entity state remained absent for the full 30-second polling timeout.This is not control-queue poison-message exhaustion.
DurableTask.AzureStoragehas no maximum dequeue count that deletes or abandons a control message. Abandoned messages back off to a maximum 10-minute visibility delay and continue retrying; reaching that delay only emits a poison-message log. In this failure, each early-delivery iteration creates a new queue message, so its dequeue count starts over.Expected behavior
A scheduled entity operation should remain pending until its due time and then execute exactly once, even when the storage emulator exposes the queue message slightly early.
Possible direction
Investigate the final due-time transition in
TaskEntityDispatcher.DetermineWorkand the entity scheduler's duplicate/message-sorter state. Separately, round a positive sub-second remaining duration up to a nonzero queue visibility delay, or otherwise prevent the immediate early-delivery loop under Azurite.Reproduction
Run:
The issue is timing-dependent; the full E2E suite increases reproduction frequency.