Skip to content

Add in-memory causal ordering for task run recorder - #18634

Merged
desertaxle merged 5 commits into
mainfrom
in-memory-causal-ordering
Aug 5, 2025
Merged

desertaxle merged 5 commits into
mainfrom
in-memory-causal-ordering

Conversation

@desertaxle

@desertaxle desertaxle commented Aug 1, 2025 •

Copy link
Copy Markdown
Member

Causal event ordering was removed from the task run recorder in #15244, which I firmly believe has led to the errors seen in #17767. Task run events may arrive out of order. While it's rare, with enough volume, out-of-order task run events become common and can lead to failed inserts and tasks appearing to stay in running if the COMPLETED races the initial event for a task and loses.

The original causal ordering implementation used the database for storage, which led to poor DB performance. This PR introduces an in-memory implementation of causal ordering to avoid DB performance degradation while getting the benefits of causal ordering for single-server setups.

For multi-server setups, a causal ordering implementation already exists in prefect-redis. We can add a mechanism to swap out the causal ordering implementation in a subsequent PR.

Closes #17767

@codspeed

codspeed Bot commented Aug 1, 2025 •

Copy link
Copy Markdown

CodSpeed Performance Report

Merging #18634 will not alter performance

Comparing in-memory-causal-ordering (905d142) with main (e825a0f)

Summary

✅ 2 untouched benchmarks

@github-actions github-actions Bot added the bug Something isn't working label Aug 1, 2025

@zzstoatzz zzstoatzz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome! just a couple nits

Comment thread src/prefect/server/events/ordering/memory.py Outdated
Comment thread src/prefect/server/events/ordering/memory.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prefect server experiencing deadlocks with Postgres database

2 participants