Skip to content

[To Dev/1.3]Encountering empty tablets during batch export of tsfile does not affect subsequent export tasks #14463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CritasWang merged 11 commits into from
Dec 18, 2024
Merged

[To Dev/1.3]Encountering empty tablets during batch export of tsfile does not affect subsequent export tasks #14463

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2820671
Update export script prompt information (#14276)
2b3c511 Dec 5, 2024
6ed522d
Merge remote-tracking branch 'origin/dev/1.3' into dev/1.3
Dec 6, 2024
4197a20
Merge remote-tracking branch 'origin/dev/1.3' into dev/1.3
Dec 11, 2024
b6f1076
Merge remote-tracking branch 'origin/dev/1.3' into dev/1.3
Dec 17, 2024
13d7b1f
Merge remote-tracking branch 'origin/dev/1.3' into dev/1.3
Dec 17, 2024
c455e96
Encountering empty tablets during batch export of tsfile does not aff…
Dec 17, 2024
f5826b9
remove warn prompt when the tablet to be exported is empty
Dec 17, 2024
4813d5d
remove warn prompt when the tablet to be exported is empty
Dec 17, 2024
b532ce8
remove warn prompt when the tablet to be exported is empty
Dec 17, 2024
dd96c12
format
Dec 17, 2024
38df95d
format
Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions integration-test/src/test/java/org/apache/iotdb/tools/it/ExportTsFileTestIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ protected void testOnWindows() throws IOException {
"exit",
"%^errorlevel%");
builder.environment().put("CLASSPATH", libPath);
testOutput(builder, output, 1);
testOutput(builder, output, 0);

prepareData();

Expand Down Expand Up @@ -148,7 +148,7 @@ protected void testOnUnix() throws IOException {
"-q",
"select * from root.**");
builder.environment().put("CLASSPATH", libPath);
testOutput(builder, output, 1);
testOutput(builder, output, 0);

prepareData();

Expand Down
31 changes: 18 additions & 13 deletions iotdb-client/cli/src/main/java/org/apache/iotdb/tool/tsfile/ExportTsFile.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,9 +337,11 @@ private static void dumpResult(String sql, int index) {
final String path = targetDirectory + targetFile + index + ".tsfile";
try (SessionDataSet sessionDataSet = session.executeQueryStatement(sql, timeout)) {
long start = System.currentTimeMillis();
writeWithTablets(sessionDataSet, path);
long end = System.currentTimeMillis();
ioTPrinter.println("Export completely!cost: " + (end - start) + " ms.");
boolean isComplete = writeWithTablets(sessionDataSet, path);
if (isComplete) {
long end = System.currentTimeMillis();
ioTPrinter.println("Export completely!cost: " + (end - start) + " ms.");
}
} catch (StatementExecutionException
| IoTDBConnectionException
| IOException
Expand Down Expand Up @@ -460,7 +462,7 @@ private static void writeWithTablets(
"squid:S3776",
"squid:S6541"
}) // Suppress high Cognitive Complexity warning, Suppress many task in one method warning
public static void writeWithTablets(SessionDataSet sessionDataSet, String filePath)
public static Boolean writeWithTablets(SessionDataSet sessionDataSet, String filePath)
throws IOException,
IoTDBConnectionException,
StatementExecutionException,
Expand All @@ -471,7 +473,7 @@ public static void writeWithTablets(SessionDataSet sessionDataSet, String filePa
if (f.exists()) {
Files.delete(f.toPath());
}

boolean isEmpty = false;
try (TsFileWriter tsFileWriter = new TsFileWriter(f)) {
// device -> column indices in columnNames
Map<String, List<Integer>> deviceColumnIndices = new HashMap<>();
Expand All @@ -483,16 +485,19 @@ public static void writeWithTablets(SessionDataSet sessionDataSet, String filePa

List<Tablet> tabletList = constructTablets(deviceSchemaMap, alignedDevices, tsFileWriter);

if (tabletList.isEmpty()) {
ioTPrinter.println("!!!Warning:Tablet is empty,no data can be exported.");
System.exit(CODE_ERROR);
if (!tabletList.isEmpty()) {
writeWithTablets(
sessionDataSet, tabletList, alignedDevices, tsFileWriter, deviceColumnIndices);
tsFileWriter.flushAllChunkGroups();
} else {
isEmpty = true;
}

writeWithTablets(
sessionDataSet, tabletList, alignedDevices, tsFileWriter, deviceColumnIndices);

tsFileWriter.flushAllChunkGroups();
}
if (isEmpty) {
ioTPrinter.println("!!!Warning:Tablet is empty,no data can be exported.");
return false;
}
return true;
}

private static void writeToTsFile(
Expand Down