Fix the infinite waiting for shutdown due to throttler limit

[wenbingshen]

Descriptions of the changes in this PR:

Motivation

If the compactor is limited, the shutdown priority should be higher than waiting for RateLimiter.acquire.

Changes

According to @hangc0276 suggestion, when processing the shutdown logic of GarbageCollectorThread, we should check the status of the newScanner.process method. If the status is false, throw an IOException and stop compact immediately.

Master Issue: #2941

[wenbingshen]

@eolivelli @merlimat @codelipenghui Can you take a look at this pr. Thanks.

[hangc0276]

@wenbingshen Thanks for you enormous contribution.
The root cause of preventing GarbageCollectorThread shutdown is that the compacting flag only control one compact entry phase.

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java

Lines 558 to 579 in 7385d90

	protected void compactEntryLog(EntryLogMetadata entryLogMeta) {
	// Similar with Sync Thread
	// try to mark compacting flag to make sure it would not be interrupted
	// by shutdown during compaction. otherwise it will receive
	// ClosedByInterruptException which may cause index file & entry logger
	// closed and corrupted.
	if (!compacting.compareAndSet(false, true)) {
	// set compacting flag failed, means compacting is true now
	// indicates that compaction is in progress for this EntryLogId.
	return;
	}
	try {
	// Do the actual compaction
	compactor.compact(entryLogMeta);
	} catch (Exception e) {
	LOG.error("Failed to compact entry log {} due to unexpected error", entryLogMeta.getEntryLogId(), e);
	} finally {
	// Mark compaction done
	compacting.set(false);
	}
	}

So, it will wait for one entry log file compact complete before GarbageCollectorThread shutdown. If we throttle the compaction in low throughput, it will take multi minutes to compaction 2GB entry log file.

IMO, we'd better introduce a new compact flag for AbstractLogCompactor and check the status on newScanner.process method, if the status is false, just throw an IOException, and stop the compact immediately. It can work for both EntryLogCompactor and TransactionalEntryLogCompactor.

[wenbingshen]

@hangc0276 Gotcha! Thanks for your review. I will implement it soon. :)

[wenbingshen]

rerun failure checks

[wenbingshen]

rerun failure checks

[wenbingshen]

rerun failure checks

[wenbingshen]

rerun failure checks

[wenbingshen]

@hangc0276 PTAL, thanks. :)

[dlg99]

LGTM

[wenbingshen]

ping @eolivelli PTAL, Thanks.

[wenbingshen]

@eolivelli PTAL Thanks

[wenbingshen]

@pkumar-singh It's been a long time. Can you take a look at this pr? Thanks.

[wenbingshen]

@eolivelli It's been over 3 months since this PR was created. Can you help take a look at this PR? Thanks very much!

[hangc0276]

@wenbingshen Would you please rebase the master? thanks.

[hangc0276]

Good job!

[hangc0276]

https://github.com/apache/bookkeeper/runs/7519602108?check_suite_focus=true
The CI failed, Please take a look, thanks.