Investigate optimal / configurable batch sizes for accumulo-gc deletion candidates

[ctubbsii]

@Manno15 's PR #1650 employed a batching strategy of fixed size batches. This is quite nice for consistent memory utilization, and ensuring the accumulo-gc can always make progress. However, the hard-coded 8MB batch size may not be the optimal size.

Investigation is needed to determine what batch sizes might optimize throughput, and whether it makes sense to make the batch size configurable. Different batch sizes should be tested.

[Manno15]

I'll take a look at this today.

[ctubbsii]

I'll take a look at this today.

Cool. Thanks!

[Manno15]

My preliminary test results are showing that very minimal time can be saved from changing the batch size to a larger value. I plan on writing a more comprehensive test tomorrow to get a clearer conclusion.

[ctubbsii]

I'm curious if it matters whether the candidates still have references in the metadata table or not, for the overall throughput. For example, if a batch size of 8MB results in most candidates being still in use, and only a few are actually available for deletion, does the performance drop on the DFS call to actually delete the file? Would it be better to have a larger batch size in order to make that later phase more efficient after the in-use candidates have been removed?

Perhaps you can provide some timing information for a few scenarios. If none of the scenarios seem to be substantially different in the throughput, it's probably not worth making a new property to make this configurable.

[EdColeman]

I may be just adding more words and Christopher's comment has it covered, but in case this helps.

It would be interesting to know some of the ratios when you run different timing scenarios - like number of candidates vs the number of deletes.

I think the thing we are trying to gauge is if there is a bottle neck in the Accumulo gc processor and do different batch sizes have any impact on where / when a bottle neck could be triggered? Something that could help determine that, is there a timing difference when there is a large number of candidate and few deletes vs a large number of candidates and lots of deletes? The first case may provide insight into the Accumulo overhead, while the second could be dominated by hdfs.
The overall goal is to determine if there is are situations where Accumulo could get into a state where it just cannot keep up with deletes and would fall further and further behind. If that can be shown, then what are the triggering conditions and where are the bottle necks, and does batch size have any impact?

Some of this could be simulated, but there also needs to measurements that include hdfs interactions. If it can be shown that the Accumulo gc process is never a bottle neck regardless of batch size, then we know to focus on hdfs interactions for additional performance improvements. If there are times where the gc process is dominating the gc cycle, then what is the performance difference with different batch sizes?

[ctubbsii]

Feel free to check in to the Slack channel if you have questions. I've been trying to be available there more often when I'm working, leaving the video call open for anybody to jump in if they want.

[Manno15]

Have you two used the gcs testing suite in accumulo-testing? I am wondering if I can adapt it to test for this.

[EdColeman]

No, I am not familiar with that suite.

[milleruntime]

@Manno15 I was also curious about that. I haven't used it yet but Keith did write up some instructions here: https://github.com/apache/accumulo-testing/blob/master/docs/gcs.md

[Manno15]

Yeah, I looked through the docs and ran though the functions a bit. I might be able to create a performance test based on it.

[ctubbsii]

Yeah, I looked through the docs and ran though the functions a bit. I might be able to create a performance test based on it.

That'd be cool.