Skip to content

Commit

Permalink
Merge pull request #12 from WenchaoZeng/useTempTables
Browse files Browse the repository at this point in the history 
Use temp tables
  • Loading branch information
@WenchaoZeng
WenchaoZeng committed Nov 30, 2023
2 parents c34bf13 + 79a5622 commit 8c0d30b
Show file tree
Hide file tree
Showing 13 changed files with 303 additions and 50 deletions.
22 changes: 17 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,11 @@ select 1 as a;

# 数据库配置文件

在当前目录下, 文件名称为: `databases.json`. 首次运行时会自动生成这个文件的默认内容, 默认自带一个 [h2](http://www.h2database.com/html/commands.html) 内存数据库, 名称为 `h2`, 可以做简单的测试和使用.
在当前目录下, 文件名称为: `databases.json`. 首次运行时会自动生成这个文件的默认内容, 默认自带一个 [h2](http://www.h2database.com/html/commands.html) 内存数据库, 名称为 `h2`, 可以做简单的测试和使用. 推荐自己配置一个mysql数据库使用.

# SQL文件结构和语法定义

文件后缀建议为sql, 以让文本编辑器支持sql语法的高亮显示.
文件后缀建议为sql, 以让你的文本编辑器支持sql语法的高亮显示.

## `# import` 导入一个xls, xlsx或csv文件

Expand Down Expand Up @@ -115,7 +115,7 @@ from test_table temp;

## 在SQL中可以使用 `$xxx` 的方式来引用之前的结果集

注意: 一定要在结果集后面加一个自己的自定义名称, 结果集是以一个sub query的方式运行的, sql标准要求一定要指定一个别名的.
注意: 一定要在结果集后面加一个自己的自定义名称, 结果集是以一个子查询(或临时表)的方式运行的, sql标准要求子查询一定要指定一个别名.

```sql
# import /Users/xxx/Downloads/xxx.xlsx
Expand Down Expand Up @@ -189,6 +189,18 @@ from test_table2
;
```

## 关于空值null的导出显示控制
## 空值null的导出显示控制

默认情况下, 值为null会在导出时候显示成空字符串. 可以使用 `# export nulls` 来指定在导出文件的时候, null被导出为`<null>`, 这样可以看出来具体哪个是null值, 区分与空字符串. 然后, 后续可以使用 `# no export nulls` 来恢复默认行为, 即null被导出为空白, 和空字符串显示一样.
默认情况下, 值为null会在导出时候显示成空字符串. 可以使用`# exportnulls`来指定在导出文件的时候, null被导出为`<null>`, 这样可以看出来具体哪个是null值, 与空字符串作区分. 可以使用`# -exportnulls`来恢复默认行为, 即null被导出为空白, 和空字符串显示一样.

## 导出xlsx格式

使用`# exportxlsx`设置xlsx导出格式, 使用`# -exportxlsx`关闭xlsx导出格式. 默认: 关闭xlsx导出格式, 即导出格式为csv.

## 临时表模式

默认情况下, 如果一个sql里引用了结果集, 结果集会以子查询的方式嵌入到sql中. 如果结果集数据量很大, 会导致子查询的sql很大, 从而导致超出java里String的最大容量, 或者超出mysql库的最大sql长度.

使用临时表模式, 可以预先把结果集的数据导入到临时表中, 然后sql中直接从临时表查询结果集, 这样sql就比较短.

使用`# temptables`打开临时表模式, 使用`# -temptables`关闭临时表模式. 默认: 关闭临时表模式.
7 changes: 7 additions & 0 deletions note.txt
View file
Original file line number Diff line number Diff line change
@@ -1 +1,8 @@
待开发功能:
针对常见的错误, 直接在控制台显示具体的错误, 不需要打印调用栈
运行输出: 控制台+文件输出.
子查询模式下, 检查子查询的长度(只有第一个select才需要指定列名称, 后续select不需要执行列名称)

代码重构:
Sql, 根据不同的语句类型, 拆分成不同的子类.
使用实例化的方式来调用工具类, 防止方法参数过多的问题.
13 changes: 6 additions & 7 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,9 @@
</dependency>

<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>3.10-FINAL</version>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
<version>4.4</version>
</dependency>

<dependency>
Expand All @@ -89,13 +89,12 @@
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>3.10-FINAL</version>
<version>5.2.2</version>
</dependency>

<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml-schemas</artifactId>
<version>3.10-FINAL</version>
<artifactId>poi</artifactId>
<version>5.2.2</version>
</dependency>

<dependency>
Expand Down
10 changes: 6 additions & 4 deletions src/main/java/com/zwc/sqldataprocessor/ExportExecutor.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,23 @@

import com.zwc.sqldataprocessor.entity.DataList;
import com.zwc.sqldataprocessor.exporter.CsvExporter;
import com.zwc.sqldataprocessor.exporter.Exporter;
import com.zwc.sqldataprocessor.exporter.XlsxExporter;

public class ExportExecutor {
public static String export(String resultName, DataList table, String path, boolean exportNulls) {
public static String export(String resultName, DataList table, String path, boolean exportNulls, boolean exportXlsx) {

// 执行导出
CsvExporter exporter = new CsvExporter();
Exporter exporter = exportXlsx ? new XlsxExporter() : new CsvExporter();
byte[] bytes = exporter.export(table, exportNulls);

// 写入导出文件
if (path == null) {
path = String.format("./%s.csv", resultName);
path = String.format("./%s.%s", resultName, exporter.getExtension());
path = FileHelper.writeOutputFile(path, bytes);
return path;
} else if (!path.contains("/") && !path.contains("\\")) {
path = String.format("./%s.csv", path);
path = String.format("./%s.%s", path, exporter.getExtension());
path = FileHelper.writeOutputFile(path, bytes);
return path;
}
Expand Down
102 changes: 81 additions & 21 deletions src/main/java/com/zwc/sqldataprocessor/SqlExecutor.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,25 @@
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

import com.zwc.sqldataprocessor.entity.DataList;
import com.zwc.sqldataprocessor.entity.DataList.ColumnType;
import org.apache.commons.lang3.StringUtils;

public class SqlExecutor {

public static DataList exec(String sql, String databaseName, Map<String, DataList> tables) {
String rawSql = renderSql(sql, tables, databaseName);
public static DataList exec(String sql, String databaseName, Map<String, DataList> tables, boolean useTempTables) {
String rawSql = renderSql(sql, tables, databaseName, useTempTables);

// 解除group_concat的长度限制
if (DatabaseConfigLoader.isMySql(databaseName)) {
execRawSql("SET SESSION group_concat_max_len = 1000000;", databaseName);
}

return execRawSql(rawSql, databaseName);
}

Expand All @@ -28,13 +36,6 @@ public static DataList execRawSql(String sql, String databaseName) {

Connection conn = DatabaseConfigLoader.getConn(databaseName);

// 解除group_concat的长度限制
if (DatabaseConfigLoader.isMySql(databaseName)) {
try (Statement cmd = conn.createStatement()) {
cmd.execute("SET SESSION group_concat_max_len = 1000000;");
}
}

try (Statement cmd = conn.createStatement()) {
boolean hasResult = cmd.execute(sql);

Expand Down Expand Up @@ -116,7 +117,8 @@ static DataList readResult(ResultSet resultSet) throws SQLException {
return table;
}

static String renderSql(String sql, Map<String, DataList> tables, String databaseName) {
static String renderSql(String sql, Map<String, DataList> tables, String databaseName, boolean useTempTables) {
Set<String> createdTempTables = new HashSet<>();
StringBuilder sqlBuilder = new StringBuilder();
for (String sqlLine : sql.split("\n")) {

Expand All @@ -143,20 +145,42 @@ static String renderSql(String sql, Map<String, DataList> tables, String databas
throw new RuntimeException("结果集$" + tableName + "不存在.");
}

// 构建数据集sql语句
StringBuilder builder = new StringBuilder();
builder.append("(\n");
String tableSelectSql;
if (DatabaseConfigLoader.isH2(databaseName)) {
tableSelectSql = renderSelectSqlForH2(table);
} else {
tableSelectSql = renderSelectSql(table);
String tableReplacement = "";
if (useTempTables) { // 构建临时表
String tempTableName = "_temp_" + tableName.replace("$", "");

if (!createdTempTables.contains(tempTableName)) {
String sqlFormat = "drop temporary table if exists %s;";
if (DatabaseConfigLoader.isH2(databaseName)) {
sqlFormat = "drop table if exists %s;";
}
execRawSql(String.format(sqlFormat, tempTableName), databaseName);
String createTempTableSql = renderCreateTempTableSql(table, tempTableName, databaseName);
execRawSql(createTempTableSql, databaseName);

// 分批导入数据
List<DataList> dataLists = table.split(10000);
for (DataList dataList : dataLists) {
String dataInsertSql = "insert into " + tempTableName + " ";
dataInsertSql += renderSelectSql(dataList, databaseName);
execRawSql(dataInsertSql, databaseName);
}

createdTempTables.add(tempTableName);
}

tableReplacement = tempTableName + " ";
} else { // 构建数据集sql语句
StringBuilder builder = new StringBuilder();
builder.append("(\n");
String tableSelectSql = renderSelectSql(table, databaseName);
builder.append(tableSelectSql);
builder.append(") ");
tableReplacement = builder.toString();
}
builder.append(tableSelectSql);
builder.append(") ");

// 替换到原sql里
sqlLine = sqlLine.replace("$" + tableName + " ", builder.toString());
sqlLine = sqlLine.replace("$" + tableName + " ", tableReplacement);
}

sqlBuilder.append(sqlLine);
Expand All @@ -166,6 +190,42 @@ static String renderSql(String sql, Map<String, DataList> tables, String databas
return sqlBuilder.toString();
}

static String renderCreateTempTableSql(DataList table, String tableName, String databaseName) {
StringBuilder builder = new StringBuilder();
builder.append("create temporary table " + tableName + "(");
for (int index = 0; index < table.columns.size(); ++index) {
builder.append("`" + table.columns.get(index) + "` ");

ColumnType columnType = table.columnTypes.get(index);
if (columnType == ColumnType.INT) {
builder.append("bigint");
} else if (columnType == ColumnType.DECIMAL) {
builder.append("decimal");
} else if (columnType == ColumnType.DATETIME) {
builder.append("datetime");
} else {
builder.append("LONGTEXT");
}

if (index != table.columns.size() - 1) {
builder.append(",");
}
}
builder.append(") ");
if (DatabaseConfigLoader.isMySql(databaseName)) {
builder.append("collate utf8mb4_general_ci CHARACTER SET utf8mb4");
}
return builder.toString();
}

static String renderSelectSql(DataList table, String databaseName) {
if (DatabaseConfigLoader.isH2(databaseName)) {
return renderSelectSqlForH2(table);
} else {
return renderSelectSql(table);
}
}

static String renderSelectSql(DataList table) {
if (table.rows.size() <= 0) {
return renderEmptySelectSql(table);
Expand Down
8 changes: 4 additions & 4 deletions src/main/java/com/zwc/sqldataprocessor/SqlFileExecutor.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,15 @@ static void internalExec(String filePath, Consumer<String> logPrinter) {
if (sql.type == SqlType.SQL) {
logPrinter.accept("SQL: " + sql.databaseName);
logPrinter.accept(sql.sql);
dataList = SqlExecutor.exec(sql.sql, sql.databaseName, tables);
dataList = SqlExecutor.exec(sql.sql, sql.databaseName, tables, sql.useTempTables);
lastResultName = sql.resultName != null ? sql.resultName : defaultResultName;
tables.put(lastResultName, dataList);
printSqlStatus(lastResultName, dataList, logPrinter, startTime);
}

// 导出
if (sql.type == SqlType.EXPORT || sql.type == SqlType.END) {
doExport(lastResultName, dataList, logPrinter, sql.fileName, sql.exportNulls);
doExport(lastResultName, dataList, logPrinter, sql.fileName, sql.exportNulls, sql.exportXlsx);

// 提前结束
if (sql.type == SqlType.END) {
Expand All @@ -75,10 +75,10 @@ static void internalExec(String filePath, Consumer<String> logPrinter) {
}
}

static void doExport(String resultName, DataList dataList, Consumer<String> logPrinter, String filePath, boolean exportNulls) {
static void doExport(String resultName, DataList dataList, Consumer<String> logPrinter, String filePath, boolean exportNulls, boolean exportXlsx) {
// 导出
logPrinter.accept("导出结果集: " + resultName);
String exportPath = ExportExecutor.export(resultName, dataList, filePath, exportNulls);
String exportPath = ExportExecutor.export(resultName, dataList, filePath, exportNulls, exportXlsx);
exportPath = Paths.get(exportPath).toFile().getAbsolutePath();
logPrinter.accept("导出文件路径: " + exportPath);

Expand Down
34 changes: 30 additions & 4 deletions src/main/java/com/zwc/sqldataprocessor/SqlLoader.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,25 +13,48 @@ public static List<Sql> loadSql(String filePath) {
String fileContent = FileHelper.readFile(filePath);
List<Sql> sqlList = new ArrayList<>();
boolean exportNulls = false;
boolean exportXlsx = false;
boolean useTempTables = false;
for (String line : fileContent.split("\n")) {

if (line.startsWith("# ")) {
line = line.trim();
String lowerLine = line.toLowerCase();

if (line.startsWith("# end")) {
if (lowerLine.equals("# end")) {
Sql endSql = new Sql();
endSql.type = SqlType.END;
sqlList.add(endSql);
continue;
}

if (line.startsWith("# no export nulls")) {
if (lowerLine.equals("# exportnulls")) {
exportNulls = true;
continue;
}

if (lowerLine.equals("# -exportnulls")) {
exportNulls = false;
continue;
}

if (line.startsWith("# export nulls")) {
exportNulls = true;
if (lowerLine.equals("# exportxlsx")) {
exportXlsx = true;
continue;
}

if (lowerLine.equals("# -exportxlsx")) {
exportXlsx = false;
continue;
}

if (lowerLine.equals("# temptables")) {
useTempTables = true;
continue;
}

if (lowerLine.equals("# -temptables")) {
useTempTables = false;
continue;
}

Expand All @@ -53,6 +76,7 @@ public static List<Sql> loadSql(String filePath) {
Sql exportSql = new Sql();
exportSql.type = SqlType.EXPORT;
exportSql.exportNulls = exportNulls;
exportSql.exportXlsx = exportXlsx;
String exportFilePath = line.replace("# export", "").trim();
if (!exportFilePath.equals("")) {
exportSql.fileName = exportFilePath;
Expand All @@ -71,6 +95,7 @@ public static List<Sql> loadSql(String filePath) {
sql.databaseName = databaseName;
sql.sql = "";
sql.resultName = getResultName(line);
sql.useTempTables = useTempTables;
sqlList.add(sql);
continue;
}
Expand All @@ -92,6 +117,7 @@ public static List<Sql> loadSql(String filePath) {
Sql exportSql = new Sql();
exportSql.type = SqlType.EXPORT;
exportSql.exportNulls = exportNulls;
exportSql.exportXlsx = exportXlsx;
sqlList.add(exportSql);
}

Expand Down
Loading

0 comments on commit 8c0d30b

Please sign in to comment.