Skip to content

Commit

Permalink
[CARBONDATA-3481] Multi-thread pruning fails when datamaps count is j…
Browse files Browse the repository at this point in the history
…ust near numOfThreadsForPruning

Cause : When the datamaps count is just near numOfThreadsForPruning,
As code is checking '>= ', last thread may not get the datamaps for prune.
Hence array out of index exception is thrown in this scenario.
There is no issues with higher number of datamaps.

Solution: In this scenario launch threads based on the distribution value,
not on the hardcoded value

This closes #3336
  • Loading branch information
ajantha-bhat authored and kunal642 committed Aug 1, 2019
1 parent 10f3747 commit 765712a
Showing 1 changed file with 9 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,6 @@ private List<ExtendedBlocklet> pruneMultiThread(List<Segment> segments,
*/

int numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning();
LOG.info(
"Number of threads selected for multi-thread block pruning is " + numOfThreadsForPruning
+ ". total files: " + totalFiles + ". total segments: " + segments.size());
int filesPerEachThread = totalFiles / numOfThreadsForPruning;
int prev;
int filesCount = 0;
Expand Down Expand Up @@ -254,6 +251,15 @@ private List<ExtendedBlocklet> pruneMultiThread(List<Segment> segments,
// this should not happen
throw new RuntimeException(" not all the files processed ");
}
if (datamapListForEachThread.size() < numOfThreadsForPruning) {
// If the total datamaps fitted in lesser number of threads than numOfThreadsForPruning.
// Launch only that many threads where datamaps are fitted while grouping.
LOG.info("Datamaps is distributed in " + datamapListForEachThread.size() + " threads");
numOfThreadsForPruning = datamapListForEachThread.size();
}
LOG.info(
"Number of threads selected for multi-thread block pruning is " + numOfThreadsForPruning
+ ". total files: " + totalFiles + ". total segments: " + segments.size());
List<Future<Void>> results = new ArrayList<>(numOfThreadsForPruning);
final Map<Segment, List<ExtendedBlocklet>> prunedBlockletMap =
new ConcurrentHashMap<>(segments.size());
Expand Down

0 comments on commit 765712a

Please sign in to comment.