Durable Entities - State cleared / lost by framework

[kweebtronic]

Description

My dotnet-isolated durable entities are randomly losing state - the entity state class being reset to the default constructor values.

Background
I use durable entities within my application, to independently track high-level and detailed summary state of a suite of durable functions. These have been reliable since durable entities was first introduced.

I have now attempted the major upgrade to dotnet-8 and isolated functions, after waiting for durable entities to reach parity.

I use a class-based entity that implements TaskEntity<TState>, as this was the closest approximation to the previously-recommended pattern for strongly-typed proxies, not currently implemented under dotnet-isolated

Expected behavior

Durable entity retains its state until deleted

Actual behavior

State is reset back to the 'Initialiser' / class default constructor (retains EntityID)

Relevant source code snippets

Repro repo - https://github.com/kweebtronic/durable-entity-state-loss
Failure state occurs most times when run on local machine

Known workarounds

Can not recover from a cleared entity. This is a critical failure

App Details

• Durable Functions extension version: Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.1.1
• Azure Functions runtime version: ~4
• Programming language used: dotnet 8 (isolated)

Screenshots

N/A

If deployed to Azure

Can be replicated independently of Azure, this appears to be a logic bug.

[jacekkulis]

Any updates or input from MS team? I belive there are more ppl affected after migration to .net8 and isolated worker...

[cgillum]

@sebastianburckhardt is this something you're aware of?

[ZTonks]

Can confirm to have seen the same issue - entities will seemingly very nondeterministically lose state and reset to C# default. Fairly critical

[kweebtronic]

Looks like a fix has been merged to main for DurableTask.Core
Azure/durabletask#1080 (comment)
Assuming it just needs time now to be rippled out to the inheriting worker extensions

[stevenheinen]

We have this same issue as well after migrating to isolated worker.
We've tried the new DurableTask.Core release (2.17.0) with the supposed fix, but that did not fix it

[wahyuen]

for others tracking this issue, the latest 'fix' has caused larger regressions. MS Support have indicated this feedback has been put back to the product group and are looking for a fix next release. I would highly avoid https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.DurableTask/1.1.3 this version as we had a complete failure (hangs, does not process at all with critical internal failure) of Durable Entities in this version.

[nytian]

Hi, @kweebtronic . Sorry for the trouble and inconvenience. The issue is caused by the inappropriate error handling. And I created a custom ADO feed to help mitigate the issue right now.

If you are interested, the ADO feed is here. To use this, you need to first connect with feed, detailed instructions can be found here : first adding package source durabletask-test in your project's nuget.config, and then run command dotnet restore. After successfully connected, you can use the feed directly in your project. Please let me know if you have any question regarding this. Thank you!

And official release might take several days, but we will make the fix as official release as soon as possible and keep you updated.

[nytian]

Hi, @kweebtronic the package with the hotfixes has been released. Please use the package v1.1.4 instead to mitigate the issue. Nuget package link: https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.Extensions.DurableTask/1.1.4
And please let me know if there is any problem.

[kweebtronic]

Fantastic work @nytian! This one looks like a winner. Thanks for your help 👍