[C++] If an I/O error is encountered while the dataset writer is applying backpressure there could be deadlock in rare (artifical?) situations

[asfimport]

When I/O errors occur the dataset writer handles them when the next batch would be written (or at teardown time if it was the last batch).

If the dataset writer has applied backpressure then this doesn't work because there is no "next batch".

This was encountered in the python backpressure test which is a bit artificial. To recreate this change the GatingFs::open_output_stream method to return None after its unlocked.

This is fairly minor because it doesn't really happen in practice. It takes many concurrent writes to get into backpressure and unless all of the fail there will be some that succeed and eventually get the writer out of backpressure enough to discover the failure.

Still, it would be nice to fix this to clean things up.

Reporter: Weston Pace / @westonpace

_{Note: This issue was originally created as ARROW-16259. Please see the migration documentation for further details.}

[asfimport]

Eduardo Ponce / @edponce:
Is there a proposed solution for this?

[asfimport]

Weston Pace / @westonpace:
Probably the easiest thing to do is when the failure occurs the file writer should take whatever rows it has reserved from the backpressure and release them. The only tricky part is that the failure is going to pop up from an I/O thread and so we need to make sure we're properly guarding access to the backpressure to avoid race conditions.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.