Add Storage Cleanup Logic - #2123
Merged
Merged
Conversation
sadeemsaleh
marked this pull request as draft
August 22, 2023 22:22
shengquan-ni
requested changes
Aug 24, 2023
shengquan-ni
left a comment
shengquan-ni
left a comment
Contributor
There was a problem hiding this comment.
left some comments.
shengquan-ni
self-requested a review
September 5, 2023 12:23
shengquan-ni
approved these changes
Sep 5, 2023
shengquan-ni
left a comment
shengquan-ni
left a comment
Contributor
There was a problem hiding this comment.
LGTM. will leave the refactoring in a separate PR.
shengquan-ni
added a commit
that referenced
this pull request
Sep 26, 2023
yangzhang75
pushed a commit
to yangzhang75/texera
that referenced
this pull request
Jun 22, 2026
This PR adds logic to clean mongodb results. Below are the changes to this PR and a description of the lifecycle of the cleanup. 1. assign the execution number to every result so that the results of different executions are persisted, 2. before compiling the logical DAG to the physical DAG, we store metadata of an execution in MySQL with pointers of where the Mongo collections will be stored, 3. when we start the server for the first time, we do a one time cleanup to delete the results of those workflows that were killed/aborted , we can identify these workflows if they have a status other than completed because there is no way in our current engine that the workflows are still running/paused etc when the server is just initialized, 4. then we start a recurring function that periodically deletes expired results based on a configured Time-to-Live, 5. the function checks MySQL's execution table that already includes the status, pointers to the results, first time, last update time, etc., 6. when we delete a result, we also update the pointers in MySQL so that they point to no results, to avoid checking them again. Co-authored-by: Shengquan Ni <13672781+shengquan-ni@users.noreply.github.com>
yangzhang75
pushed a commit
to yangzhang75/texera
that referenced
this pull request
Jun 22, 2026
After apache#2031, the logic in apache#2123 for updating mongo collection pointers is missing when we create sink storages. This PR added the missing code and should fix apache#2161.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds logic to clean mongodb results. Below are the changes to this PR and a description of the lifecycle of the cleanup.