Permalink
Show file tree
Hide file tree
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Better handle errors when something goes wrong writing partitions (#154)
* Better handle errors when something goes wrong writing partitions Follow up to #108. There were actually a few different issues all coming together in @kjanosz's comment. The first being that `ZipFile.Reader` doesn't like non-main threads touching its files _at all_. The Arrow.jl problem there is when processing non-first partitions, we were just `Threads.@spawn`-ing a task which then went off and sometimes became the proverbial unheard tree falling in the forest. Like really, no one heard these tasks and their poor exceptions. The solution there is to be better shepherds of our spawned tasks; we introduce a `anyerror` atomic bool that threads can set if they run into issues and then in the main partition handling loop we'll check that. If a thread ran into something, we'll log out the thread-specific exception, then throw a generic writing error. This prevents the "hanging" behavior people were seeing because the felled threads were actually causing later tasks to hang on `put!`-ing into the OrderedChannel. After addressing this, we have the multithreaded ZipFile issue. With the new `ntasks` keyword, it seems to make sense to me that if you pass `ntasks=1`, you're really saying, I don't want any concurrency. So anywhere we were `Threads.@spawn`ing, we now check if `ntasks == 1` and if so, do an `@async` instead. * Fix
- Loading branch information
Showing
3 changed files
with
41 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters