Close inlined savers on exceptions in multiprocessing #390
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the problem / what does the code in this PR do
This is a fix for exception handling during multiprocessing. Previously, if strax crashed during multiprocessing,
_temp
directories could remain on disk, and exceptions were not always recorded in metadata. This should fix that. It's a low-priority PR: this won't prevent any crashes, just handle them slightly nicer. In other words, a perfect hobby project for a 'retired' colleague ;-).The problem in more detail:
ParallelSourcePlugin
"inlines" some savers, so the save operation can happen in the same process that processes the data. Inlined savers are outside the mailbox system, and do not get informed of exceptions the usual way.The PR looks bigger than it is; it changes many lines just to indent them.
iter
won't win any beauty contests with its high indentation level; if you see a good way to refactor it, go for it!Can you briefly describe how it works?
Two changes were needed to fix this:
Plugin.iter
so thecleanup
code is always called: basically, we wrap everything in a big try-except loop. This also serves another purpose: replacing duplicated 'cleanup + return' calls with raising a custom exception.Mailbox._send_from
so that when sending to a mailbox fails -- most importantly because the mailbox is killed -- wethrow
the exception back to the sending iterable, i.e. the plugin or loader producing the input. Usually nothing will happen with this, and the exception is just reraised. However, it givesParallelSourcePlugin
a chance to call cleanup. This will wait for any outstanding save operations and close its savers (i.e. consolidate metadata, record the exception, and remove _temp). Finally, the exception will still be reraised, and strax continues to die as per usual.Can you give a minimal working example (or illustrate with a figure)?
Try adding
rechunk_on_save = False
toRecords
in testutils.py; this will cause master to failtest_exception
intest_core
.