GEODE-8029: Allow OplogEntryIdSet to Overflow

[jujoramos]

Do not delete drf files during member startup as that should be only
done by the compactor thread. Instead, allow the OplogEntryIdSet to
grow over the default capacity and log a warning message instructing
the user to manually compact the disk-stores.

• Added unit tests.
• Replaced usages of 'junit.Assert' by 'assertj'.
• Modified DiskStoreImpl.deadRecordCount to return long instead of int.
• Added internal overflow implementation to the OplogEntryIdSet so it can
  grow above the default limit.

Thank you for submitting a contribution to Apache Geode.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

• Has your PR been rebased against the latest commit within the target branch (typically develop)?

• Is your initial contribution a single, squashed commit?

• Does gradlew build run cleanly?

• Have you written or updated unit tests to verify your changes?

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:

Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

[dschneider-pivotal]

This looks like a good quick fix for users hitting this issue. I'm concerned though that if they are having this issue that a large amount of memory will be used to track deleted ids. An int[] of size 805,306,401 consumes 3.2G of memory.
We should consider a more long term fix. We used to have code that would remove drfs that were no longer needed. If we could do that more periodically it seems like it would prevent a large number of drfs from building up.
Also it seems like the OplogEntryIdSet itself could be reimplemented to use less memory. Since the ids for an diskstore always start at 1 and then simply increase, and many use cases will end up with large sequences of ids that have been deleted, we could consider using algorithms like we have on our concurrency check code for tracking what version ids we have seen. Basically you can just have a sequence (start of sequence and length) which can represent a long sequence with just 2 32-bit values (or 2 64-bit values if longs). Anything in the sequence range that is still alive you would note in another data structure as an exception. If exceptions are rare then this can save lots of memory.

[pivotal-eshu]

If we no longer need the OplogEntryIdSet after the recovery, shall we clear these sets once it is done?

With these many entryIds in drf files, it will also take longer time to recover. I think Darrel's idea for removing unnecessary drf files will help in this case as well.

[jujoramos]

@pivotal-eshu

If we no longer need the OplogEntryIdSet after the recovery, shall we clear these sets once it is done?

There are no references to the OplogEntryIdSet instance once we finish the recovery, so the class itself (along with all internal structures) will eventually be removed by the garbage collector.

With these many entryIds in drf files, it will also take longer time to recover. I think Darrel's idea for removing unnecessary drf files will help in this case as well.

Agreed, I'll create another ticket for that.