This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Better handle errors when something goes wrong writing partitions (#154)
* Better handle errors when something goes wrong writing partitions Follow up to #108. There were actually a few different issues all coming together in @kjanosz's comment. The first being that `ZipFile.Reader` doesn't like non-main threads touching its files _at all_. The Arrow.jl problem there is when processing non-first partitions, we were just `Threads.@spawn`-ing a task which then went off and sometimes became the proverbial unheard tree falling in the forest. Like really, no one heard these tasks and their poor exceptions. The solution there is to be better shepherds of our spawned tasks; we introduce a `anyerror` atomic bool that threads can set if they run into issues and then in the main partition handling loop we'll check that. If a thread ran into something, we'll log out the thread-specific exception, then throw a generic writing error. This prevents the "hanging" behavior people were seeing because the felled threads were actually causing later tasks to hang on `put!`-ing into the OrderedChannel. After addressing this, we have the multithreaded ZipFile issue. With the new `ntasks` keyword, it seems to make sense to me that if you pass `ntasks=1`, you're really saying, I don't want any concurrency. So anywhere we were `Threads.@spawn`ing, we now check if `ntasks == 1` and if so, do an `@async` instead. * Fix
- Loading branch information
Showing 3 changed files with 41 additions and 13 deletions.