Durable Entities #184

davidmrdavid · 2020-08-25T06:21:37Z

Addresses: #96

Durable Entities for Python

This PR enables Durable Entities for Python, enabling our users to manage small pieces of state, the Durable way.

At a high-level, this PR includes a handful of changes:

It introduces a whole set of Entity utility classes, in the /models/entities/ directory, which help to encapsulate the different kinds of objects that the durable-extension returns. For the most part, these resemble their DurableJS counterparts.
It specifies the a syntax for declaring Entities in python, which you can find in our new Counter-based sample.
It introduces a handful of TODO comments over the codebase, based on observations I made while studying the DurableJS implementation of Entities. I want to keep these TODOs as they specify areas of the codebase that need more thought and design, but that are left as-is because they match the behaviour of JS. Admittedly, some other ToDos may just be actually leftover tasks.
Introduces a very simple, barebones, unit test for Durable Entities. We'll want to test this feature more throughly than it currently is, but this PR takes a first stab. In the future, we'll want to the not just the DurableOrchestrationContext as it makes calls to Durable Entities, but also a DurableEntityContext as it receives mocked requests from orchestrators.

Finally, these changes are dependent on changes on the Python worker that have not been merged yet. Those changes are in this small PR: Azure/azure-functions-python-library#68 . In any case, we do not want to merge this until that PR is merged.

davidmrdavid · 2020-08-25T06:57:51Z

Bug found:
Ah, I just found one bug where I'm not looking up correctly the result of a callEntity in the state array. Oh well, that's one for tomorrow :)

davidmrdavid · 2020-08-27T02:41:42Z

Bug fixed 🛩️

ConnorMcMahon

Very good start to this.

Since we have a bit of time before this merges, I would like to see some units tests for the entity itself instead of just the call_entity API for the orchestration. I would also like to see some basic signals working if at all possible.

azure/durable_functions/models/actions/CallEntityAction.py

azure/durable_functions/entity.py

azure/durable_functions/models/DurableEntityContext.py

ConnorMcMahon · 2020-09-09T23:12:03Z

azure/durable_functions/models/DurableEntityContext.py

+        bool
+            True if the Entity was newly constructed. False otherwise.
+        """
+        # TODO: not updating this atm


Not sure if this even gets updated when C# handles a batch either, so this may not be necessary. We should consult Sebastian about this.

@sebastianburckhardt:

Is there a use for a isNewlyConstructed flag for entities? Or can it be safely removed?

I'll be tracking this here: #222

azure/durable_functions/models/DurableEntityContext.py

ConnorMcMahon · 2020-09-09T23:14:14Z

azure/durable_functions/models/DurableEntityContext.py

+        self._exists = True
+        self._result = result
+
+    def destruct_on_exit(self) -> None:


I'm not sure how we handle this in JS or C#, but there is the question of how to handle subsequent operations after this within a batch.

Let's check-in with the team then 😉
@cgillum , @sebastianburckhardt:

What are the semantics for a destructed entity (via destructOnExit) when there are still operations to perform on a batch?

I'll be tracking this here: #222

azure/durable_functions/models/utils/entity_utils.py

azure/durable_functions/tasks/task_utilities.py

azure/durable_functions/models/actions/CallEntityAction.py

Signal Entity

* add client signal entity support * modify the formating

It looked the brackets []() are hyper-link

…ctions-durable-python into dajusto/entities

davidmrdavid · 2020-11-13T20:31:25Z

I believe this is ready for merge!

Co-authored-by: cvitzthum-nasuni <cvitzthum@nasuni.com>

@kemurayama

Create links for sample folder. Co-authored with @kemurayama

* Replace datetime.strptime * Deleted unnecessary format string * Delete whitespace

ConnorMcMahon

Mostly nits, but a few of these I would want addressed before we merge.

Overall, exciting stuff though!

azure/durable_functions/entity.py

azure/durable_functions/models/DurableEntityContext.py

azure/durable_functions/models/entities/EntityState.py

azure/durable_functions/models/utils/entity_utils.py

azure/durable_functions/orchestrator.py

…ctions-durable-python into dajusto/entities

ConnorMcMahon

LGTM!

davidmrdavid added 8 commits August 9, 2020 14:48

created helper classes

0249145

merge and clean-up

87b7640

typechecking entity-context

e24f8a6

moving entity-context

4f27d2b

Durable entities are functional, linted, typechecked

cf8d86a

added simple test

402b9ad

added sample, missing test

14844b3

removed leftover print statement

e69b2b5

davidmrdavid marked this pull request as ready for review August 25, 2020 06:34

davidmrdavid mentioned this pull request Aug 25, 2020

Added support for Entity Trigger Azure/azure-functions-python-library#68

Merged

hidden object references are removed

98db559

davidmrdavid requested a review from ConnorMcMahon August 27, 2020 17:12

ConnorMcMahon reviewed Sep 9, 2020

View reviewed changes

davidmrdavid added 2 commits September 11, 2020 16:35

PR feedback

0247d32

merge

b2c3707

davidmrdavid commented Sep 11, 2020

View reviewed changes

azure/durable_functions/models/actions/CallEntityAction.py Outdated Show resolved Hide resolved

davidmrdavid and others added 6 commits September 11, 2020 16:45

minor change in CallEntityAction to overcomposate for GitHub acting up

b513535

linting

023fd70

Rewind API (#163)

65188f2

Implemented out-of-proc error reporting schema (#196)

6c25a78

support signal entity

1881257

unit test and fixes in test infra

a30f63b

wenhzha mentioned this pull request Oct 6, 2020

Signal Entity #206

Merged

wenhzha and others added 3 commits October 12, 2020 14:13

lint

33fb7ca

lint and fix test

7f350a2

Merge pull request #206 from Azure/wenhzha/signal-entity

69ec826

Signal Entity

davidmrdavid mentioned this pull request Oct 17, 2020

Add support for Durable Entities #96

Closed

Added missing 'kind' field in TokenSource's serialized form (#209)

f047698

davidmrdavid and others added 5 commits October 20, 2020 13:46

disabling schema validation (#212)

59e7708

Add Client Signal Entity (#216)

dc90330

* add client signal entity support * modify the formating

Create link for sample folder

0d9a327

It looked the brackets []() are hyper-link

Merge branch 'dajusto/entities' of https://github.com/Azure/azure-fun…

9c48eb0

…ctions-durable-python into dajusto/entities

added entitycontext, fixed sample

4bb82dc

davidmrdavid mentioned this pull request Nov 13, 2020

Durable Entities ToDos #222

Open

2 tasks

davidmrdavid added 2 commits November 13, 2020 11:55

serialize custom types

5105a54

sample uses signal

4d6f22b

davidmrdavid requested a review from ConnorMcMahon November 18, 2020 18:14

carlvitzthum and others added 4 commits November 18, 2020 15:39

Serialize input for CallActivityWithRetryAction (#225)

af482d2

Co-authored-by: cvitzthum-nasuni <cvitzthum@nasuni.com>

Merge branch 'dev' into dev

523de06

Merge pull request #219 from kemurayama/dev

0f8679b

Create links for sample folder. Co-authored with @kemurayama

Replace datetime.strptime for find_task_timer_created() (#228)

d95c56d

* Replace datetime.strptime * Deleted unnecessary format string * Delete whitespace

ConnorMcMahon reviewed Dec 3, 2020

View reviewed changes

davidmrdavid added 2 commits December 2, 2020 22:37

Merge branch 'dajusto/entities' of https://github.com/Azure/azure-fun…

4241369

…ctions-durable-python into dajusto/entities

PR feedback

85192e5

davidmrdavid requested a review from ConnorMcMahon December 3, 2020 17:12

This was referenced Dec 3, 2020

Improve static safety of repo by using generic types and custom constructors #231

Open

Improve naming conventions in Entity implementation #232

Open

ConnorMcMahon approved these changes Dec 3, 2020

View reviewed changes

not handling internal exceptions, deferring to extension

274d4c0

davidmrdavid mentioned this pull request Dec 3, 2020

Ensure Internal entity exceptions are correctly handled #233

Open

davidmrdavid merged commit 46a0505 into dev Dec 3, 2020

davidmrdavid deleted the dajusto/entities branch December 3, 2020 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Durable Entities #184

Durable Entities #184

davidmrdavid commented Aug 25, 2020 •

edited

Loading

davidmrdavid commented Aug 25, 2020

davidmrdavid commented Aug 27, 2020

ConnorMcMahon left a comment

ConnorMcMahon Sep 9, 2020

davidmrdavid Sep 11, 2020

davidmrdavid Nov 13, 2020

ConnorMcMahon Sep 9, 2020

davidmrdavid Sep 11, 2020

davidmrdavid Nov 13, 2020

davidmrdavid commented Nov 13, 2020

ConnorMcMahon left a comment

ConnorMcMahon left a comment

Durable Entities #184

Durable Entities #184

Conversation

davidmrdavid commented Aug 25, 2020 • edited Loading

davidmrdavid commented Aug 25, 2020

davidmrdavid commented Aug 27, 2020

ConnorMcMahon left a comment

Choose a reason for hiding this comment

ConnorMcMahon Sep 9, 2020

Choose a reason for hiding this comment

davidmrdavid Sep 11, 2020

Choose a reason for hiding this comment

davidmrdavid Nov 13, 2020

Choose a reason for hiding this comment

ConnorMcMahon Sep 9, 2020

Choose a reason for hiding this comment

davidmrdavid Sep 11, 2020

Choose a reason for hiding this comment

davidmrdavid Nov 13, 2020

Choose a reason for hiding this comment

davidmrdavid commented Nov 13, 2020

ConnorMcMahon left a comment

Choose a reason for hiding this comment

ConnorMcMahon left a comment

Choose a reason for hiding this comment

davidmrdavid commented Aug 25, 2020 •

edited

Loading