Skip to content

Commit

Permalink
[CARBONDATA-2571] Calculating the carbonindex and carbondata file siz…
Browse files Browse the repository at this point in the history
…e of a table is wrong

Problem:
While calculating the carbonindex files size, we are checking either index file or merge file. But in PR#2333, implementation is changed to fill both
the file name and the merge file name. So, we have to consider both fields.

Solution:
While calculating the carbonindex files size, we have to consider both the files and mergeFileName fields. We should get the list of index files from
these 2 fields and then calculate the size of the files.

This closes #2358
  • Loading branch information
dhatchayani authored and manishgupta88 committed Jun 5, 2018
1 parent 92d9b92 commit 27d7059
Showing 1 changed file with 20 additions and 17 deletions.
37 changes: 20 additions & 17 deletions core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
Original file line number Diff line number Diff line change
Expand Up @@ -2688,27 +2688,30 @@ private static HashMap<String, Long> getDataSizeAndIndexSize(SegmentFileStore fi
throws IOException {
long carbonDataSize = 0L;
long carbonIndexSize = 0L;
List<String> listOfFilesRead = new ArrayList<>();
HashMap<String, Long> dataAndIndexSize = new HashMap<String, Long>();
if (fileStore.getLocationMap() != null) {
Map<String, SegmentFileStore.FolderDetails> locationMap = fileStore.getLocationMap();
if (locationMap != null) {
fileStore.readIndexFiles();
Map<String, String> indexFiles = fileStore.getIndexFiles();
Map<String, List<String>> indexFilesMap = fileStore.getIndexFilesMap();
for (Map.Entry<String, List<String>> entry : indexFilesMap.entrySet()) {
// get the size of carbonindex file
String indexFile = entry.getKey();
String mergeIndexFile = indexFiles.get(indexFile);
if (null != mergeIndexFile) {
String mergeIndexPath = indexFile
.substring(0, indexFile.lastIndexOf(CarbonCommonConstants.FILE_SEPARATOR) + 1)
+ mergeIndexFile;
if (!listOfFilesRead.contains(mergeIndexPath)) {
carbonIndexSize += FileFactory.getCarbonFile(mergeIndexPath).getSize();
listOfFilesRead.add(mergeIndexPath);
}
} else {
carbonIndexSize += FileFactory.getCarbonFile(indexFile).getSize();
// get the size of carbonindex file
for (Map.Entry<String, SegmentFileStore.FolderDetails> entry : locationMap.entrySet()) {
SegmentFileStore.FolderDetails folderDetails = entry.getValue();
Set<String> carbonindexFiles = folderDetails.getFiles();
String mergeFileName = folderDetails.getMergeFileName();
if (null != mergeFileName) {
String mergeIndexPath =
fileStore.getTablePath() + entry.getKey() + CarbonCommonConstants.FILE_SEPARATOR
+ mergeFileName;
carbonIndexSize += FileFactory.getCarbonFile(mergeIndexPath).getSize();
}
for (String indexFile : carbonindexFiles) {
String indexPath =
fileStore.getTablePath() + entry.getKey() + CarbonCommonConstants.FILE_SEPARATOR
+ indexFile;
carbonIndexSize += FileFactory.getCarbonFile(indexPath).getSize();
}
}
for (Map.Entry<String, List<String>> entry : indexFilesMap.entrySet()) {
// get the size of carbondata files
for (String blockFile : entry.getValue()) {
carbonDataSize += FileFactory.getCarbonFile(blockFile).getSize();
Expand Down

0 comments on commit 27d7059

Please sign in to comment.