Skip to content

Commit

Permalink
[CARBONDATA-2805] Fix the ordering mismatch of segment numbers during…
Browse files Browse the repository at this point in the history
… cutom compaction

Problem:
when we have segments from 0 to 6 and i give 1, 2, 3 for custom compaction, then it should create 1.1 as compacted segment, but sometimes
it will create 3.1 as compacted segment which is wrong. This is beacuse custom Segment IDs were passing in hashset and finally inserted in
hashmap, while identifying segments to be merged. hashmap and hashset does not guarantee the insertion order which may lead to missmatch of segment numbers.

Solution:
Use LinkedHashSet and LinkedHashMap which always sure about the insertion order.

This closes apache#2585
  • Loading branch information
akashrn5 authored and sgururajshetty committed Aug 2, 2018
1 parent ec73f91 commit 6fc4980
Showing 1 changed file with 4 additions and 6 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,6 @@ public static List<LoadMetadataDetails> identifySegmentsToBeMerged(
CarbonLoadModel carbonLoadModel, long compactionSize,
List<LoadMetadataDetails> segments, CompactionType compactionType,
List<String> customSegmentIds) throws IOException, MalformedCarbonCommandException {
String tablePath = carbonLoadModel.getTablePath();
Map<String, String> tableLevelProperties = carbonLoadModel.getCarbonDataLoadSchema()
.getCarbonTable().getTableInfo().getFactTable().getTableProperties();
List<LoadMetadataDetails> sortedSegments = new ArrayList<LoadMetadataDetails>(segments);
Expand All @@ -400,7 +399,7 @@ public static List<LoadMetadataDetails> identifySegmentsToBeMerged(

if (CompactionType.CUSTOM == compactionType) {
return identitySegmentsToBeMergedBasedOnSpecifiedSegments(sortedSegments,
new HashSet<>(customSegmentIds));
new LinkedHashSet<>(customSegmentIds));
}

// Check for segments which are qualified for IUD compaction.
Expand All @@ -424,7 +423,7 @@ public static List<LoadMetadataDetails> identifySegmentsToBeMerged(
if (CompactionType.MAJOR == compactionType) {

listOfSegmentsToBeMerged = identifySegmentsToBeMergedBasedOnSize(compactionSize,
listOfSegmentsLoadedInSameDateInterval, carbonLoadModel, tablePath);
listOfSegmentsLoadedInSameDateInterval, carbonLoadModel);
} else {

listOfSegmentsToBeMerged =
Expand Down Expand Up @@ -462,7 +461,7 @@ private static List<LoadMetadataDetails> identitySegmentsToBeMergedBasedOnSpecif
List<LoadMetadataDetails> listOfSegments,
Set<String> segmentIds) throws MalformedCarbonCommandException {
Map<String, LoadMetadataDetails> specifiedSegments =
new HashMap<>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
new LinkedHashMap<>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
for (LoadMetadataDetails detail : listOfSegments) {
if (segmentIds.contains(detail.getLoadName())) {
specifiedSegments.put(detail.getLoadName(), detail);
Expand Down Expand Up @@ -623,13 +622,12 @@ private static boolean isTwoDatesPresentInRequiredRange(Date segDate1, Date segD
* @param listOfSegmentsAfterPreserve the segments list after
* preserving the configured number of latest loads
* @param carbonLoadModel carbon load model
* @param tablePath the store location of the segment
* @return the list of segments that need to be merged
* based on the Size in case of Major compaction
*/
private static List<LoadMetadataDetails> identifySegmentsToBeMergedBasedOnSize(
long compactionSize, List<LoadMetadataDetails> listOfSegmentsAfterPreserve,
CarbonLoadModel carbonLoadModel, String tablePath) throws IOException {
CarbonLoadModel carbonLoadModel) throws IOException {

List<LoadMetadataDetails> segmentsToBeMerged =
new ArrayList<>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
Expand Down

0 comments on commit 6fc4980

Please sign in to comment.