Skip to content

Possible event loss when using raise_event on suspended orchestration #42

Open
@cgillum

Description

@cgillum

The CI has been a bit flakey lately.

Here was the output of a recent tests/test_orchestration_e2e.py::test_suspend_and_resume test.

2025-02-13 23:18:15.166 durabletask-worker INFO: Starting gRPC worker that connects to localhost:4001
2025-02-13 23:18:15.166 durabletask-client INFO: Starting new 'orchestrator' instance with ID = '6a0f557f863f41bba1d78f6fffaaa15f'.
2025-02-13 23:18:15.167 durabletask-worker INFO: Successfully connected to localhost:4001. Waiting for work items...
2025-02-13 23:18:15.167 durabletask-client INFO: Waiting up to 30s for instance '6a0f557f863f41bba1d78f6fffaaa15f' to start.
2025-02-13 23:18:15.181 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Orchestrator yielded with 0 task(s) and 1 event(s) outstanding.
2025-02-13 23:18:15.183 durabletask-client INFO: Suspending instance '6a0f557f863f41bba1d78f6fffaaa15f'.
2025-02-13 23:18:15.230 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Execution suspended.
2025-02-13 23:18:15.230 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Orchestrator yielded with 0 task(s) and 1 event(s) outstanding.
2025-02-13 23:18:15.284 durabletask-client INFO: Raising event 'my_event' for instance '6a0f557f863f41bba1d78f6fffaaa15f'.
2025-02-13 23:18:15.285 durabletask-client INFO: Waiting 3s for instance '6a0f557f863f41bba1d78f6fffaaa15f' to complete.
2025-02-13 23:18:15.337 durabletask-worker INFO: 6a0f557f863f41bba1d78f6fffaaa15f: Orchestrator yielded with 0 task(s) and 1 event(s) outstanding.
2025-02-13 23:18:18.286 durabletask-worker INFO: Stopping gRPC worker...
2025-02-13 23:18:18.287 durabletask-worker INFO: Disconnected from localhost:4001
2025-02-13 23:18:18.287 durabletask-worker INFO: No longer listening for work items
2025-02-13 23:18:18.287 durabletask-worker INFO: Worker shutdown completed

It's possible that this could be related to the backend, but it's unclear. Either way this needs to be investigated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions