Skip to content

Commit

Permalink
Merge pull request #33 from IlyaLisov/#32
Browse files Browse the repository at this point in the history 
#32 Add migration property to skip or pass rows by rules
  • Loading branch information
@IlyaLisov
IlyaLisov committed Jun 30, 2023
2 parents 5ebd94b + 5acaabd commit 7f52286
Show file tree
Hide file tree
Showing 18 changed files with 308 additions and 149 deletions.
59 changes: 47 additions & 12 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ pass `XML_VALIDATION_ENABLED` property as `true`.
<newName>newName</newName>
</columns>
</renamedColumns>
<follow>
<column value="OneType">dtype</column>
</follow>
<skip>
<column value="AnotherType">dtype</column>
</skip>
<timeFormat>yyyy-MM-dd'T'HH:mm:ss.SSS'Z'</timeFormat>
</configuration>
<labels>
Expand All @@ -66,6 +72,12 @@ pass `XML_VALIDATION_ENABLED` property as `true`.
<sourceLabel>User</sourceLabel>
<targetColumn>task_id</targetColumn>
<targetLabel>Task</targetLabel>
<follow>
<column value="OneType">dtype</column>
</follow>
<skip>
<column value="AnotherType">dtype</column>
</skip>
</configuration>
<type>HAS_TASK</type>
</table>
Expand Down Expand Up @@ -102,32 +114,44 @@ pass `XML_VALIDATION_ENABLED` property as `true`.
8) `<column>` - column tag, contains table name.
9) `<renamedColumns>` - (optional for `node` migration) columns to be renamed
during migration. It means data from column with `<previousName>`
will be stored as `<newName>` property;
10) `<timeFormat>` - (optional for `node` migration) format of timestamp to
will be stored as `<newName>` property.
10) `<follow>` - (optional for `node`, `relationship` migration) follows only
that rows which cells in these columns are equal to `value` attribute
of `<column>` tag. If multiple columns are provided, all columns match is
required to migrate it.
11) `<skip>` - (optional for `node`, `relationship` migration) skips only
that rows which cells in these columns are equal to `value` attribute
of `<column>` tag. If multiple columns are provided, at least one match is
required to skip it.
12) `<timeFormat>` - (optional for `node` migration) format of timestamp to
store in Neo4j. It is needed to store LocalDateTime and access it without
converters in code.
11) `<labels>` - (optional for `node` migration) collection of labels to be
13) `<labels>` - (optional for `node` migration) collection of labels to be
added
to Nodes.
12) `<label>` - label tag, defines its name.
13) `<sourceColumn>` - column with foreign key to entity table. Relationship will
14) `<label>` - label tag, defines its name.
15) `<sourceColumn>` - column with foreign key to entity table. Relationship
will
be started from Node from that table by this foreign key. Inner field will
be added to node with this primary key.
14) `<targetColumn>` - column with foreign key to entity table. Relationship will
16) `<targetColumn>` - column with foreign key to entity table. Relationship
will
be ended with Node from that table by this foreign key.
15) `<sourceLabel>` - (optional for `migration`, `innerField` migration) specifies
17) `<sourceLabel>` - (optional for `relationship`, `innerField` migration)
specifies
label of
start
node to find it by
foreign key.
16) `<targetLabel>` - (optional for `migration` migration) specifies label of end
18) `<targetLabel>` - (optional for `relationship` migration) specifies label of
end
node to
find it by foreign
key.
17) `<type>` - type of the relationship.
18) `<valueColumn>` - name of column with value for inner field migration.
19) `<fieldName>` - name of inner field of node to set value to.
20) `<unique>` - (optional for `innerField` migration) specify whether values in
19) `<type>` - type of the relationship.
20) `<valueColumn>` - name of column with value for inner field migration.
21) `<fieldName>` - name of inner field of node to set value to.
22) `<unique>` - (optional for `innerField` migration) specify whether values in
inner field must be unique. False if not present.

### NOTE
Expand All @@ -144,13 +168,24 @@ Note that at first we exclude columns and only after rename them. So if you will
rename excluded columns, it was excluded and no columns with this name will be
renamed.

We first handle `<skip>` rows, it means if row matches `<skip>` rule and it
matches `<follow>` rule, it won`t be migrated.

By providing several values for one column, they are considered as array of
available values. If this array contains cell value, `<skip>` rule will skip
this row, `<follow>` rule will follow this row.

We handle Postgres types in generated JSON the following way:

- `integer`, `bigserial`, `biginteger` are considered numeric values
- `bool`, `boolean` are considered boolean values
- `timestamp`, `timestamp without time zone` as timestamp
- other types - strings.

If there are `null` in cell, we store this as `null` too, so it won`t be saved
to
node.

We recommend to fill up all tags to be sure that correct data will
be saved to Neo4j.

Expand Down
81 changes: 60 additions & 21 deletions src/main/java/com/example/postgresneo4jmigrationtool/generator/dumper/CSVPostgresDumper.java
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
package com.example.postgresneo4jmigrationtool.generator.dumper;

import com.example.postgresneo4jmigrationtool.model.DumpResult;
import com.example.postgresneo4jmigrationtool.model.MigrationData;
import com.example.postgresneo4jmigrationtool.model.exception.MigrationException;
import com.example.postgresneo4jmigrationtool.repository.postgres.PostgresRepository;
import lombok.RequiredArgsConstructor;
Expand All @@ -14,6 +14,7 @@
import java.io.InputStream;
import java.io.PrintWriter;
import java.util.Collection;
import java.util.List;
import java.util.Map;

@Service
Expand All @@ -28,55 +29,93 @@ public class CSVPostgresDumper implements PostgresDumper {
private String delimiter;

@Override
public DumpResult dump(String tableName, Collection<String> columnsToDump) {
DumpResult dumpResult = new DumpResult();
dumpResult.add("dumpDirectory", dumpDirectory);
public MigrationData dump(String tableName, Collection<String> columnsToDump, MigrationData params) {
MigrationData migrationData = new MigrationData();
migrationData.add("dumpDirectory", dumpDirectory);
File dumpScript = new File(dumpDirectory + "/" + dumpScriptFileName);
createFile(dumpScript);
String columns = String.join(",", columnsToDump);
Map<String, List<String>> followRows = (Map<String, List<String>>) params.get("followRows");
Map<String, List<String>> skipRows = (Map<String, List<String>>) params.get("skipRows");
String follow = "true = true";
if (!followRows.isEmpty()) {
for (String key : followRows.keySet()) {
List<String> values = followRows.get(key);
String valuesString = "'" + String.join("', '", values) + "'";
follow += String.format(" AND %s IN (%s)", key, valuesString);
}
}
String skip = "true = true";
if (!followRows.isEmpty()) {
for (String key : skipRows.keySet()) {
List<String> values = skipRows.get(key);
String valuesString = "'" + String.join("', '", values) + "'";
skip += String.format(" AND %s NOT IN (%s)", key, valuesString);
}
}
try (PrintWriter writer = new PrintWriter(dumpScript)) {
writer.printf("psql -U %s -c \"COPY (SELECT %s FROM %s) TO STDOUT WITH CSV DELIMITER '%s' HEADER\" %s > %s.csv",
postgresRepository.getUsername(), columns, tableName, delimiter,
writer.printf("psql -U %s -c \"COPY (SELECT %s FROM %s WHERE %s AND %s) TO STDOUT WITH CSV DELIMITER '%s' HEADER\" %s > %s.csv",
postgresRepository.getUsername(), columns, tableName, follow, skip, delimiter,
postgresRepository.getDatabaseName(), tableName);
} catch (IOException e) {
throw new MigrationException("Exception during dumping: " + e.getMessage());
}
runScript(dumpScript);
addInputStream(dumpResult, tableName);
return dumpResult;
addInputStream(migrationData, tableName);
return migrationData;
}

@Override
public DumpResult dumpWithForeignKeys(String tableName, String columnFrom, String columnTo) {
DumpResult dumpResult = new DumpResult();
dumpResult.add("dumpDirectory", dumpDirectory);
public MigrationData dumpWithForeignKeys(String tableName, String columnFrom, String columnTo, MigrationData params) {
MigrationData migrationData = new MigrationData();
migrationData.add("dumpDirectory", dumpDirectory);
File dumpScript = new File(dumpDirectory + "/" + dumpScriptFileName);
createFile(dumpScript);
String foreignColumnFrom = postgresRepository.getForeignColumnName(tableName, columnFrom);
String foreignColumnTo = postgresRepository.getForeignColumnName(tableName, columnTo);
Map<String, List<String>> followRows = (Map<String, List<String>>) params.get("followRows");
Map<String, List<String>> skipRows = (Map<String, List<String>>) params.get("skipRows");
String follow = "true = true";
if (!followRows.isEmpty()) {
for (String key : followRows.keySet()) {
List<String> values = followRows.get(key);
String valuesString = "'" + String.join("', '", values) + "'";
follow += String.format(" AND %s IN (%s)", key, valuesString);
}
}
String skip = "true = true";
if (!followRows.isEmpty()) {
for (String key : skipRows.keySet()) {
List<String> values = skipRows.get(key);
String valuesString = "'" + String.join("', '", values) + "'";
skip += String.format(" AND %s NOT IN (%s)", key, valuesString);
}
}
try (PrintWriter writer = new PrintWriter(dumpScript)) {
writer.printf("psql -U %s -c \"COPY (SELECT %s as %s, %s as %s FROM %s) TO STDOUT WITH CSV DELIMITER '%s' HEADER\" %s > %s.csv",
writer.printf("psql -U %s -c \"COPY (SELECT %s as %s, %s as %s FROM %s WHERE %s AND %s) TO STDOUT WITH CSV DELIMITER '%s' HEADER\" %s > %s.csv",
postgresRepository.getUsername(),
columnFrom,
foreignColumnFrom,
columnTo,
foreignColumnTo,
tableName,
follow,
skip,
delimiter,
postgresRepository.getDatabaseName(),
tableName);
} catch (IOException e) {
throw new MigrationException("Exception during dumping: " + e.getMessage());
}
runScript(dumpScript);
addInputStream(dumpResult, tableName);
return dumpResult;
addInputStream(migrationData, tableName);
return migrationData;
}

@Override
public DumpResult dumpInnerFields(String tableName, String columnFrom, String valueColumn) {
DumpResult dumpResult = new DumpResult();
dumpResult.add("dumpDirectory", dumpDirectory);
public MigrationData dumpInnerFields(String tableName, String columnFrom, String valueColumn) {
MigrationData migrationData = new MigrationData();
migrationData.add("dumpDirectory", dumpDirectory);
File dumpScript = new File(dumpDirectory + "/" + dumpScriptFileName);
createFile(dumpScript);
String foreignColumnFrom = postgresRepository.getForeignColumnName(tableName, columnFrom);
Expand All @@ -94,8 +133,8 @@ public DumpResult dumpInnerFields(String tableName, String columnFrom, String va
throw new MigrationException("Exception during dumping: " + e.getMessage());
}
runScript(dumpScript);
addInputStream(dumpResult, tableName);
return dumpResult;
addInputStream(migrationData, tableName);
return migrationData;
}

private void createFile(File file) {
Expand All @@ -121,10 +160,10 @@ private void runScript(File dumpScript) {
}
}

private void addInputStream(DumpResult dumpResult, String tableName) {
private void addInputStream(MigrationData migrationData, String tableName) {
try {
InputStream inputStream = new FileInputStream(dumpDirectory + "/" + tableName + ".csv");
dumpResult.add("inputStream", inputStream);
migrationData.add("inputStream", inputStream);
} catch (FileNotFoundException e) {
throw new MigrationException("Migration script file was not found.");
}
Expand Down
8 changes: 4 additions & 4 deletions src/main/java/com/example/postgresneo4jmigrationtool/generator/dumper/PostgresDumper.java
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
package com.example.postgresneo4jmigrationtool.generator.dumper;

import com.example.postgresneo4jmigrationtool.model.DumpResult;
import com.example.postgresneo4jmigrationtool.model.MigrationData;

import java.util.Collection;

public interface PostgresDumper {

DumpResult dump(String tableName, Collection<String> columnsToDump);
MigrationData dump(String tableName, Collection<String> columnsToDump, MigrationData params);

DumpResult dumpWithForeignKeys(String tableName, String columnFrom, String columnTo);
MigrationData dumpWithForeignKeys(String tableName, String columnFrom, String columnTo, MigrationData params);

DumpResult dumpInnerFields(String tableName, String columnFrom, String valueColumn);
MigrationData dumpInnerFields(String tableName, String columnFrom, String valueColumn);

}
Loading

0 comments on commit 7f52286

Please sign in to comment.