Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert sending input as global state instead of parameter #5425

Merged
merged 6 commits into from
May 22, 2024
Merged

Conversation

sfmskywalker
Copy link
Member

@sfmskywalker sfmskywalker commented May 22, 2024

This PR fixes a concurrency issue when sending concurrent workflow execution requests to the same workflow instance with different inputs.

Originally, this change was made to add support for sending large inputs from a parent workflow to a dispatched child workflow via a service bus.

However, that doesn't work when sending multiple messages concurrently, which we are seeing when using BulkDispatchWorkflow where multiple child workflows complete close to one another, overriding the "global" input state associated with the workflow input.

This PR attempts to only store the input with the workflow instance upon instance creation, while sending input alongside the dispatch message when resuming existing ones. The assumption here being that those messages include smaller input payloads.

The correct fix will be to revise the need for associating workflow input with the workflow instance, and instead provide mechanism to stash input externally from the workflow instance and instead passing in an identifier to that input.

Fixes #5417

This commit introduces a distributed lock to the MassTransitWorkflowDispatcher to prevent concurrent updates to the workflow instance. The lock is acquired before any interaction with the workflow instance and is released after the update operation is complete.
Removed unnecessary code in the MassTransitWorkflowDispatcher.cs class, previously used to handle input and properties. Refactored to directly include input and properties in the DispatchWorkflowInstanceRequest object, increasing efficiency and readability.
@sfmskywalker sfmskywalker requested review from a team May 22, 2024 13:10
@sfmskywalker sfmskywalker changed the title Revert to send input as parameter instead of global state Revert sending input as global state instead of parameter May 22, 2024
@raymonddenhaan raymonddenhaan self-requested a review May 22, 2024 14:30
Copy link
Contributor

@raymonddenhaan raymonddenhaan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add tests for the scenario that should be fixed with this?

The IDistributedLockProvider dependency in the constructor of MassTransitWorkflowDispatcher class was removed because it was identified as unnecessary in this context. This results in less complicated code and improved maintainability.
@sfmskywalker
Copy link
Member Author

@raymonddenhaan I considered it, but my last attempt to try and reproduce a race condition using a component test didn't pan out. Perhaps its possible, but I'd need a lot more time to try and achieve that. I am open to suggestions in case you have any.

@sfmskywalker
Copy link
Member Author

Actually, I could try something similar to the WorkflowDefinitionActivityTests - it's a bit crude and doesn't guarantee the bug appears in case of a regression - but worked 99% of the times.

@raymonddenhaan
Copy link
Contributor

Let's see if we can figure out how we can test these race conditions and if we can do it with 100% certainty without costing us a lot of time and resources in the build pipeline. This PR does not have to wait for that discussion.

Added a new file `BulkDispatchWorkflowsTests.cs` that contains tests for Bulk Dispatch Workflows. Also, two new workflow files `EmployeeGreetingWorkflow.cs` and `GreetEmployeesWorkflow.cs` were added to define workflows used in the unit tests. This will help ensure the Bulk Dispatch Workflows feature is working as expected.
@sfmskywalker sfmskywalker merged commit f03c62a into main May 22, 2024
3 checks passed
@sfmskywalker sfmskywalker deleted the bug/5417 branch May 22, 2024 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Concurrency Issue in BulkDispatchWorkflows Activity Prevents Completion
2 participants