Stalled writes block forever during shutting down

[qingping209]

in the following code, the Put operation block forever.

https://gist.github.com/qingping209/356febd14f2d7db7b46212b11477ebf4

here is my reasoning:

1. the db has too many level0 files, and the Put operation blocks
2. the scheduled background compaction is supposed to clear the stop write condition once it's finished
3. before the scheduled background compaction finished, a CancelAllBackgroundWork(true) is called
4. the blocked Put is wake up and recheck the stop write condition in DelayWrite, and the stop write condition is not clear;
5. the Put block in the DelayWrite again.

A possible fix is to check the shutting_down_ variable in this line: https://github.com/facebook/rocksdb/blob/main/db/db_impl/db_impl_write.cc#L1787

and return Status::ok() at https://github.com/facebook/rocksdb/blob/main/db/db_impl/db_impl_write.cc#L1787
if shutting_down_ is set.

[qingping209]

@ajkr @siying can you take a brief look ?

[cbi42]

Hi @qingping209 , the analysis makes sense to me, would you like to contribute a fix?

[qingping209]

Hi @qingping209 , the analysis makes sense to me, would you like to contribute a fix?

sure.

[Yuval-Ariel]

Hey @cbi42 ,
Is it correct to deduce that its valid to close the db while another thread is writing to it?
Im asking because CancelAllBackgroundWork is only a part of the Close() operation.

Thanks!