HDDS-7032. RDBStore#getUpdatesSince should throw SequenceNumberNotFoundException back to the caller in order for Recon to fall back to full snapshot

[smengcl]

What changes were proposed in this pull request?

Recon expects an exception to be thrown in order to fall back to full snapshot when an OM DB delta update cannot be retrieved:

ozone/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java

Lines 456 to 478 in 5ed0e0a

	LOG.info("Obtaining delta updates from Ozone Manager");
	// Get updates from OM and apply to local Recon OM DB.
	getAndApplyDeltaUpdatesFromOM(currentSequenceNumber,
	omdbUpdatesHandler);
	// Update timestamp of successful delta updates query.
	ReconTaskStatus reconTaskStatusRecord = new ReconTaskStatus(
	OmSnapshotTaskName.OmDeltaRequest.name(),
	System.currentTimeMillis(), getCurrentOMDBSequenceNumber());
	reconTaskStatusDao.update(reconTaskStatusRecord);
	// Pass on DB update events to tasks that are listening.
	reconTaskController.consumeOMEvents(new OMUpdateEventBatch(
	omdbUpdatesHandler.getEvents()), omMetadataManager);
	} catch (InterruptedException intEx) {
	Thread.currentThread().interrupt();
	} catch (Exception e) {
	metrics.incrNumDeltaRequestsFailed();
	LOG.warn("Unable to get and apply delta updates from OM.", e);
	fullSnapshot = true;
	}
	}
	if (fullSnapshot) {

However, the current logic implies that RDBStore#getUpdatesSince will never throw SequenceNumberNotFoundException back to the client (Recon) because it is caught as IOException in try-catch. A patch earlier unintentionally caused the regression.

The solution is to restore the intended behavior by explicitly catching SequenceNumberNotFoundException and throwing it back to the client.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7032

How was this patch tested?

• Manual tested on a repro cluster.

[kerneltime]

Thanks @smengcl for finding this.

[kerneltime]

Should revert back to throwing all IOExceptions?

[smengcl]

I believe SequenceNumberNotFoundException is the only expected checked exception to be thrown anyway, judging from the method signature:

ozone/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java

Lines 280 to 281 in 9466047

	public DBUpdatesWrapper getUpdatesSince(long sequenceNumber, long limitCount)
	throws SequenceNumberNotFoundException {

[swagle]

+1

[kerneltime]

I think we might have to revisit the exception handling here.

[smengcl]

I think we might have to revisit the exception handling here.

Yeah at some point we might want to refactor it.

Thanks @swagle and @kerneltime for the review.

[adoroszlai]

Unfortunately the commit message cannot be fixed...

[kerneltime]

Unfortunately the commit message cannot be fixed...

Will keep this in mind.