Skip to content

Fix using too much memory when estimate compaction task memory #10619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HTHou merged 6 commits into from
Jan 11, 2024
Merged

Fix using too much memory when estimate compaction task memory #10619

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a6eb2a5
fix oom when estimate cross space compaction memory
shuwenwei Jul 20, 2023
9e0a05a
mvn spotless:apply
shuwenwei Jul 20, 2023
576cbe3
Merge branch 'iotdb_master' into fix-oom-when-estimate-cross-space-co…
shuwenwei Nov 8, 2023
d5de3c9
resolve conflicts
shuwenwei Nov 8, 2023
4e61563
Merge branch 'iotdb_master' into fix-oom-when-estimate-cross-space-co…
shuwenwei Dec 11, 2023
a5dd7aa
Merge branch 'iotdb_master' into fix-oom-when-estimate-cross-space-co…
shuwenwei Jan 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 31 additions & 12 deletions ...db/db/storageengine/dataregion/compaction/selector/estimator/CompactionEstimateUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,13 @@
package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.estimator;

import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
import org.apache.iotdb.tsfile.file.metadata.TimeseriesMetadata;
import org.apache.iotdb.tsfile.file.metadata.ChunkMetadata;
import org.apache.iotdb.tsfile.read.TsFileDeviceIterator;
import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
import org.apache.iotdb.tsfile.utils.Pair;

import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

Expand All @@ -48,19 +51,35 @@ public static FileInfo calculateFileInfo(TsFileSequenceReader reader) throws IOE
int maxChunkNum = 0;
int maxAlignedSeriesNumInDevice = -1;
int maxDeviceChunkNum = 0;
Map<String, List<TimeseriesMetadata>> deviceMetadata = reader.getAllTimeseriesMetadata(true);
for (Map.Entry<String, List<TimeseriesMetadata>> entry : deviceMetadata.entrySet()) {
TsFileDeviceIterator deviceIterator = reader.getAllDevicesIteratorWithIsAligned();
while (deviceIterator.hasNext()) {
int deviceChunkNum = 0;
List<TimeseriesMetadata> deviceTimeseriesMetadata = entry.getValue();
if (deviceTimeseriesMetadata.get(0).getMeasurementId().equals("")) {
// aligned device
maxAlignedSeriesNumInDevice =
Math.max(maxAlignedSeriesNumInDevice, deviceTimeseriesMetadata.size());
int alignedSeriesNumInDevice = 0;

Pair<String, Boolean> deviceWithIsAlignedPair = deviceIterator.next();
String device = deviceWithIsAlignedPair.left;
boolean isAlignedDevice = deviceWithIsAlignedPair.right;

Iterator<Map<String, List<ChunkMetadata>>> measurementChunkMetadataListMapIterator =
reader.getMeasurementChunkMetadataListMapIterator(device);
while (measurementChunkMetadataListMapIterator.hasNext()) {
Map<String, List<ChunkMetadata>> measurementChunkMetadataListMap =
measurementChunkMetadataListMapIterator.next();
if (isAlignedDevice) {
alignedSeriesNumInDevice += measurementChunkMetadataListMap.size();
}

for (Map.Entry<String, List<ChunkMetadata>> measurementChunkMetadataList :
measurementChunkMetadataListMap.entrySet()) {
int currentChunkMetadataListSize = measurementChunkMetadataList.getValue().size();
deviceChunkNum += currentChunkMetadataListSize;
totalChunkNum += currentChunkMetadataListSize;
maxChunkNum = Math.max(maxChunkNum, currentChunkMetadataListSize);
}
}
for (TimeseriesMetadata timeseriesMetadata : deviceTimeseriesMetadata) {
deviceChunkNum += timeseriesMetadata.getChunkMetadataList().size();
totalChunkNum += timeseriesMetadata.getChunkMetadataList().size();
maxChunkNum = Math.max(maxChunkNum, timeseriesMetadata.getChunkMetadataList().size());
if (isAlignedDevice) {
maxAlignedSeriesNumInDevice =
Math.max(maxAlignedSeriesNumInDevice, alignedSeriesNumInDevice);
}
maxDeviceChunkNum = Math.max(maxDeviceChunkNum, deviceChunkNum);
}
Expand Down