Skip to content

Commit

Permalink
repair: Release permit earlier when the repair_reader is done
Browse files Browse the repository at this point in the history
Consider

- 10 repair instances take all the 10 _streaming_concurrency_sem

- repair readers are done but the permits are not released since they
  are waiting for view update _registration_sem

- view updates trying to take the _streaming_concurrency_sem to make
  progress of view update so it could release _registration_sem, but it
  could not take _streaming_concurrency_sem since the 10 repair
  instances have taken them

- deadlock happens

Note, when the readers are done, i.e., reaching EOS, the repair reader
replaces the underlying (evictable) reader with an empty reader. The
empty reader is not evictable, so the resources cannot be forcibly
released.

To fix, release the permits manually as soon as the repair readers are
done even if the repair job is waiting for _registration_sem.

Fixes #14676

Closes #14677
  • Loading branch information
asias authored and denesb committed Jul 13, 2023
1 parent 6a7d980 commit 1b577e0
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions repair/row_level.cc
Expand Up @@ -361,13 +361,15 @@ repair_reader::read_mutation_fragment() {

future<> repair_reader::on_end_of_stream() noexcept {
return _reader.close().then([this] {
_permit.release_base_resources();
_reader = mutation_fragment_v1_stream(make_empty_flat_reader_v2(_schema, _permit));
_reader_handle.reset();
});
}

future<> repair_reader::close() noexcept {
return _reader.close().then([this] {
_permit.release_base_resources();
_reader_handle.reset();
});
}
Expand Down

0 comments on commit 1b577e0

Please sign in to comment.