Improving retention manager to handle segment lineage clean-up

[snleee]

1. Added the logic to handle segment lineage clean-up in the
   retention manager.
2. Added the unit test

[Jackie-Jiang]

No need to check segmentsTo for COMPLETED. We can delete the lineage entry once all the segments in segmentsFrom are already deleted (not in the current round).

[Jackie-Jiang]

We can simplify the logic as following, same for IN_PROGRESS

Suggested change

	segmentsToDelete.addAll(lineageEntry.getSegmentsFrom());
	// The lineage entry with 'COMPLETED' state can only be safely removed when both segmentFrom & segmentTo
	// are all removed from the table.
	if (Collections.disjoint(segmentsForTable, lineageEntry.getSegmentsFrom()) && Collections
	.disjoint(segmentsForTable, lineageEntry.getSegmentsTo())) {
	segmentLineage.deleteLineageEntry(lineageEntryId);
	}
	Set<String> sourceSegments = new HashSet<>(lineageEntry.getSegmentsFrom());
	sourceSegments.retainAll(segmentsForTable);
	if (sourceSegments.isEmpty()) {
	segmentLineage.deleteLineageEntry(lineageEntryId);
	} else {
	segmentsToDelete.addAll(sourceSegments);
	}

[snleee]

Changed. Please double check the logic for IN_PROGRESS as well.

[Jackie-Jiang]

LGTM