HoodieCommitArchiveLog OOM's if the number of commits/cleans is too large

[n3nash]

If a job fails to archive over 1-2 days, the (numberOfCommits - minCommits) can be really large and much more than (maxCommits - minCommits) intended to be archived. Add a logic in the HoodieCommitArchiveLog to cap the number of commits to archive as follows : min( (numberOfCommits - minCommits), (maxCommits - minCommits)).

@jianxu @kaushikd49 FYI

[jianxu]

Thanks @n3nash ! Just to 1 additional note: the cap should apply to both clean instants and commit instants.

[bvaradar]

@n3nash , @jianxu : Can you link stack-trace or heap-dump when OOM occured ?

[n3nash]

Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
	at java.io.DataOutputStream.write(DataOutputStream.java:107)
	at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
	at com.uber.hoodie.common.table.log.block.HoodieAvroDataBlock.lambda$getBytes$0(HoodieAvroDataBlock.java:92)
	at com.uber.hoodie.common.table.log.block.HoodieAvroDataBlock$$Lambda$100/1958503496.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1249)
	at com.uber.hoodie.common.table.log.block.HoodieAvroDataBlock.getBytes(HoodieAvroDataBlock.java:79)

[vinothchandar]

Not sure if I want to rush into caping yet. Can we get into why the archival failed for 1-2 days? How much memory was used, how much was needed?

[n3nash]

We know why that happened, it's because our jobs were failing due to a different reason. Ideally, we shouldn't run into this issue, but the concern was that if we do, this ends up taking a huge amount of memory on the driver (12G) and even then fails, so this is worth exploring.