DurableTaskStorageException when concurrently reading and writing large data

[andhallberg]

Description

When concurrently writing and reading large data from a durable entity, a DurableTaskStorageException with message "The specified blob does not exist" is sometimes thrown from DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest.

The exception started to appear in v2.9.5.

Expected behavior

No exception thrown when concurrently reading and writing large data from a durable entity.

Actual behavior

When concurrently writing and reading large data from a durable entity, a DurableTaskStorageException with message "The specified blob does not exist" is thrown from DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest.

Relevant source code snippets

This reproducer triggers the exception about every 10s:

using System;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;

namespace Company.FunctionApp1;

public class Functions
{
    [FunctionName(nameof(WriteToDurableEntity))]
    public async Task WriteToDurableEntity(
        // Run on startup, actual schedule irrelevant
        [TimerTrigger("0 0 0 1 * *", RunOnStartup = true)] TimerInfo timer,
        [DurableClient] IDurableEntityClient client)
    {
        var data = new byte[50_000];
        while (true)   
        {
            await client.SignalEntityAsync<ITheEntity>("theEntityKey", entity => entity.WriteData(data));
        }
    }
    
    [FunctionName(nameof(ReadFromDurableEntity))]
    public async Task ReadFromDurableEntity(
        // Run on startup, actual schedule irrelevant
        [TimerTrigger("0 0 0 1 * *", RunOnStartup = true)] TimerInfo timer,
        [DurableClient] IDurableEntityClient client)
    {
        while (true)
        {
            try
            {
                _ = (await client.ReadEntityStateAsync<TheEntity>(
                    new EntityId(nameof(TheEntity), "theEntityKey"))).EntityState;
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }
    
    [FunctionName(nameof(TheEntity))]
    public static Task Run([EntityTrigger] IDurableEntityContext ctx) => ctx.DispatchAsync<TheEntity>();
}

public interface ITheEntity
{
    void WriteData(byte[] data);
}

public class TheEntity : ITheEntity
{
    public byte[] Data { get; set; } = Array.Empty<byte>();
    
    public void WriteData(byte[] data)
    {
        Data = data;
    }
}

Stacktrace:

DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist.
 ---> Microsoft.WindowsAzure.Storage.StorageException: The specified blob does not exist.
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgr
ess`1 progressHandler, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.c
s:line 157
   at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientR
equestId) in /_/src/DurableTask.AzureStorage/TimeoutHandler.cs:line 48
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureSt
orageClient.cs:line 137


 --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureSt
orageClient.cs:line 141
   at DurableTask.AzureStorage.Storage.Blob.DownloadToStreamAsync(MemoryStream target) in /_/src/DurableTask.AzureStorage/Storage/Blob.cs:line 98
   at DurableTask.AzureStorage.MessageManager.DownloadAndDecompressAsBytesAsync(Blob blob) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 229
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.ConvertFromAsync(OrchestrationInstanceStatus orchestrationInstanceStatus, String instanceId) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 5
42
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.FetchInstanceStatusInternalAsync(String instanceId, Boolean fetchInput) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 453
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.GetStateAsync(String instanceId, String executionId, Boolean fetchInput) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 422
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.GetStateAsync(String instanceId, Boolean allExecutions, Boolean fetchInput) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 416
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.GetOrchestrationStateAsync(String instanceId, Boolean allExecutions) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1651
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.AzureStorageDurabilityProvider.RetrieveSerializedEntityState(EntityId entityId, JsonSerializerSettings serializerSettings) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\Azure
StorageDurabilityProvider.cs:line 100
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.ReadEntityStateAsync[T](DurabilityProvider provider, EntityId entityId) in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\ContextImplementations\DurableClient.cs:li
ne 559
   at Company.FunctionApp1.Functions.ReadFromDurableEntity(TimerInfo timer, IDurableEntityClient client) in C:\...\Company.FunctionApp1\Functions.cs:line 34

Known workarounds

Downgrade to v2.9.4.

App Details

• Durable Functions extension version (e.g. v1.8.3): v2.9.5
• Azure Functions runtime version (1.0 or 2.0): V4
• Programming language used: C#, net6
• Azure Function type: In-process

[andhallberg]

Any progress on this issue? We are still seeing this problem in v2.11.1.

[andyhammar]

@nytian Friendly ping, any fixes in this area? Thanks

[EntityAdam]

I am currently experiencing this issue as well. However, I do not have this problem when using an actual storage account. This occurs when using Azurite local storage emulation.

While we could point fingers and ask Azurite to fix this, I believe an underlying issue that Azure Functions STILL relies `WindowsAzure.*" version of the Azure Storage SDKs which was deprecated almost a full year ago. (3/31/2023) Azure/durabletask#982

azurite -v
3.28.0

Programming language used: C#, .NET6
Azure Function type: In-process
Core Tools Version:       4.0.5611 Commit hash: N/A +591b8aec842e333a87ea9e23ba390bb5effe0655 (64-bit)
Function Runtime Version: 4.31.1.22191

[2024-03-29T18:17:52.707Z] An error occurred while processing messages on testhubname-control-01: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist.
[2024-03-29T18:17:52.708Z]  ---> Microsoft.WindowsAzure.Storage.StorageException: The specified blob does not exist.
[2024-03-29T18:17:52.709Z]    at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
[2024-03-29T18:17:52.709Z]    at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancellationToken)
[2024-03-29T18:17:52.710Z]    at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 157
[2024-03-29T18:17:52.710Z]    at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)
[2024-03-29T18:17:52.711Z]    at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
[2024-03-29T18:17:52.711Z] Request Information
[2024-03-29T18:17:52.712Z] RequestID:66eafcb3-7631-49ca-bba1-f97ad9f4f0cb
[2024-03-29T18:17:52.712Z] RequestDate:Fri, 29 Mar 2024 14:17:52 GMT
[2024-03-29T18:17:52.713Z] StatusMessage:The specified blob does not exist.
[2024-03-29T18:17:52.713Z] ErrorCode:BlobNotFound
[2024-03-29T18:17:52.714Z] ErrorMessage:The specified blob does not exist.
RequestId:66eafcb3-7631-49ca-bba1-f97ad9f4f0cb
Time:2024-03-29T18:17:52.703Z
[2024-03-29T18:17:52.714Z]
[2024-03-29T18:17:52.715Z]    --- End of inner exception stack trace ---
[2024-03-29T18:17:52.715Z]    at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 141
[2024-03-29T18:17:52.716Z]    at DurableTask.AzureStorage.Storage.Blob.DownloadToStreamAsync(MemoryStream target) in /_/src/DurableTask.AzureStorage/Storage/Blob.cs:line 98
[2024-03-29T18:17:52.716Z]    at DurableTask.AzureStorage.MessageManager.DownloadAndDecompressAsBytesAsync(Blob blob) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 229
[2024-03-29T18:17:52.717Z]    at DurableTask.AzureStorage.MessageManager.DownloadAndDecompressAsBytesAsync(String blobName) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 210
[2024-03-29T18:17:52.717Z]    at DurableTask.AzureStorage.MessageManager.DeserializeQueueMessageAsync(QueueMessage queueMessage, String queueName) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 148
[2024-03-29T18:17:52.718Z]    at DurableTask.AzureStorage.Messaging.ControlQueue.<>c__DisplayClass11_0.<<GetMessagesAsync>b__0>d.MoveNext() in /_/src/DurableTask.AzureStorage/Messaging/ControlQueue.cs:line 108
[2024-03-29T18:17:52.718Z] : Abandoning [] message back to testhubname-control-01 and setting a visibility delay of 32ms