-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[data][fault tolerance] Do not eagerly free root RefBundles #39016
Conversation
Signed-off-by: Hao Chen <chenh1024@gmail.com>
cleaned_metadata(read_task), | ||
) | ||
], | ||
owns_blocks=True, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you set owns_blocks=False that would also equivalently prevent them from being freed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though, it sounds like we'd like to free them at the end of the workload, just not as soon as possible, so there is some subtlety here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, owns_blocks=False
is the right fix. at the end of the workload, they will be eventually freed by python GC. and their sizes are small. So it should be fine.
…ect#39016) Fix the issue that we eagerly free the root RefBundles. This bug broke fault tolerance. --------- Signed-off-by: Hao Chen <chenh1024@gmail.com>
…ect#39016) Fix the issue that we eagerly free the root RefBundles. This bug broke fault tolerance. --------- Signed-off-by: Hao Chen <chenh1024@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
…ect#39016) Fix the issue that we eagerly free the root RefBundles. This bug broke fault tolerance. --------- Signed-off-by: Hao Chen <chenh1024@gmail.com>
…ect#39016) Fix the issue that we eagerly free the root RefBundles. This bug broke fault tolerance. --------- Signed-off-by: Hao Chen <chenh1024@gmail.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>
…ect#39016) Fix the issue that we eagerly free the root RefBundles. This bug broke fault tolerance. --------- Signed-off-by: Hao Chen <chenh1024@gmail.com> Signed-off-by: Victor <vctr.y.m@example.com>
Why are these changes needed?
Fix the issue that we eagerly free the root RefBundles. This bug broke fault tolerance.
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.