[fix][broker] Make sure all inflight writes have finished before completion of compaction

[coderzc]

Motivation

The previous write may not be complete by the time compaction is completed, we should wait for all inflight writes to have finished.

#20988 (comment)

Modifications

Make sure all inflight writes have finished before going to before completion of compaction.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[codecov-commenter]

Codecov Report

Merging #21067 (32b2f86) into master (30073db) will increase coverage by 36.11%.
Report is 3 commits behind head on master.
The diff coverage is 100.00%.

@@              Coverage Diff              @@
##             master   #21067       +/-   ##
=============================================
+ Coverage     36.73%   72.84%   +36.11%     
- Complexity    12197    32177    +19980     
=============================================
  Files          1696     1885      +189     
  Lines        130039   139727     +9688     
  Branches      14187    15363     +1176     
=============================================
+ Hits          47775   101791    +54016     
+ Misses        75926    29857    -46069     
- Partials       6338     8079     +1741     

Flag	Coverage Δ
inttests	24.26% <50.00%> (+0.12%)	⬆️
systests	25.29% <0.00%> (+0.28%)	⬆️
unittests	72.08% <100.00%> (+40.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...sar/broker/authentication/SaslRoleTokenSigner.java	92.30% <100.00%> (+92.30%)	⬆️
...rg/apache/pulsar/compaction/TwoPhaseCompactor.java	77.77% <100.00%> (+9.47%)	⬆️

... and 1452 files with indirect coverage changes