Feature flink order sink #56

YangZH-v2 · 2022-07-19T11:05:15Z

Implement Flink cdc sink for LakeSoul.
This PR finishes following features:

Flink Catalog and Table API. Close LakeSoul Catalog for Flink #58
Flink CDC Writer for LakeSoul. Close Flink CDC Writer #59

Flink sink writer New meta

snapshot snapshotmanagement sparkutil createtablecommand

delete configuration from doMetaCommit

…nge file_exist_cols

sonatype-lift · 2022-07-20T07:41:27Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulFileSinkHelper.java

+  public void onProcessingTime(long timestamp) throws Exception {
+    long currentTime = this.procTimeService.getCurrentProcessingTime();
+    this.buckets.onProcessingTime(currentTime);
+    this.procTimeService.registerTimer(currentTime + this.bucketCheckInterval, this);


FutureReturnValueIgnored: Return value of methods returning Future must be checked. Ignoring returned Futures suppresses exceptions thrown from the code that completes the Future.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:28Z

src/main/java/org/apache/flink/lakeSoul/tool/FlinkUtil.java

+    }
+    bd.primaryKey(Arrays.asList(hashColumn.split(",")));
+    String partitionKeys = tableInfo.getPartitions();
+    String[] split = partitionKeys.split(";");


StringSplitter: String.split(String) has surprising behavior

Suggested change

String[] split = partitionKeys.split(";");

List<String> split = Splitter.on(';').splitToList(partitionKeys);

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:31Z

src/main/java/org/apache/flink/lakeSoul/sink/sort/LakeSoulSortTask.java

+  private transient StreamRecordCollector<RowData> collector;
+  private transient BinaryRowDataSerializer binarySerializer;
+  private HashMap<String, Integer> buckets;
+  private LakeSoulKeyGen keyGen;


UnusedVariable: The field 'keyGen' is never read.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:33Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulBucket.java

+        case INT_KEY_TYPE:
+          sortQueue = new PriorityQueue<>((v1, v2) -> {
+            try {
+              return keyGen.getSimpleIntKey(v1) - (keyGen.getSimpleIntKey(v2));


UnnecessaryParentheses: These grouping parentheses are unnecessary; it is unlikely the code will be misinterpreted without them

Suggested change

return keyGen.getSimpleIntKey(v1) - (keyGen.getSimpleIntKey(v2));

return keyGen.getSimpleIntKey(v1) - keyGen.getSimpleIntKey(v2);

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:35Z

src/main/java/org/apache/flink/lakeSoul/metaData/LakeSoulCatalog.java

+        FileSystem fileSystem = new Path(path).getFileSystem();
+        qualifiedPath = new Path(path).makeQualified(fileSystem).toString();
+      } catch (IOException e) {
+        e.printStackTrace();


CatchAndPrintStackTrace: Logging or rethrowing exceptions should usually be preferred to catching and calling printStackTrace

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:36Z

src/main/java/org/apache/flink/lakeSoul/sink/partition/DataPartitioner.java

+import org.apache.spark.unsafe.types.UTF8String;
+import static org.apache.spark.sql.types.DataTypes.StringType;
+
+public class DataPartitioner<String> implements Partitioner<String> {


TypeNameShadowing: Found type parameters shadowing other declared types:

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:38Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulBucket.java

+          break;
+      }
+    } catch (NoSuchMethodException e) {
+      e.printStackTrace();


CatchAndPrintStackTrace: Logging or rethrowing exceptions should usually be preferred to catching and calling printStackTrace

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:40Z

src/main/java/org/apache/flink/lakeSoul/sink/sort/LakeSoulKeyAndValueSerializer.java

+  }
+
+  @Override
+  public boolean equals(Object o) {


EqualsGetClass: Prefer instanceof to getClass when implementing Object#equals.

Suggested change

public boolean equals(Object o) {

if (!(o instanceof LakeSoulKeyAndValueSerializer)) {

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:42Z

src/main/java/org/apache/flink/lakeSoul/metaData/LakeSoulCatalog.java

+      throw new PartitionNotExistException(LAKE_SOUL_DATA_BASE_NAME, tablePath, catalogPartitionSpec);
+    }
+    String tableName = tablePath.getFullName();
+    String rangeValue = FlinkUtil.getRangeValue(catalogPartitionSpec);


UnusedVariable: The local variable 'rangeValue' is never read.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:43Z

src/main/java/org/apache/flink/lakeSoul/metaData/LakeSoulCatalog.java

+    }
+    String tableName = tablePath.getFullName();
+    String rangeValue = FlinkUtil.getRangeValue(catalogPartitionSpec);
+    TableInfo tableInfo = dbManager.getTableInfo(tableName);


UnusedVariable: The local variable 'tableInfo' is never read.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:44Z

src/main/java/org/apache/flink/lakeSoul/sink/sort/LakeSoulSortTask.java

+  private transient BinaryExternalSorter sorter;
+  private transient StreamRecordCollector<RowData> collector;
+  private transient BinaryRowDataSerializer binarySerializer;
+  private HashMap<String, Integer> buckets;


UnusedVariable: The field 'buckets' is never read.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:47Z

src/main/java/org/apache/flink/lakeSoul/table/LakeSoulDynamicTableFactory.java

+              FactoryUtil.FORMAT.key()));
+    }
+
+    final List<Factory> factories = new LinkedList<>();


JdkObsolete: It is very rare for LinkedList to out-perform ArrayList or ArrayDeque. Avoid it unless you're willing to invest a lot of time into benchmarking. Caveat: LinkedList supports null elements, but ArrayDeque does not.

Suggested change

final List<Factory> factories = new LinkedList<>();

final List<Factory> factories = new ArrayList<>();

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:49Z

src/main/java/org/apache/flink/lakeSoul/tool/LakeSoulKeyGen.java

+  }
+
+  public static int objToInt(@Nullable Object obj) {
+    return obj == null ? null : (int) obj;


NullTernary: This conditional expression may evaluate to null, which will result in an NPE when the result is unboxed.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:50Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulFileSinkHelper.java

+    }
+
+    long currentProcessingTime = procTimeService.getCurrentProcessingTime();
+    procTimeService.registerTimer(currentProcessingTime + bucketCheckInterval, this);


FutureReturnValueIgnored: Return value of methods returning Future must be checked. Ignoring returned Futures suppresses exceptions thrown from the code that completes the Future.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:52Z

src/main/java/org/apache/flink/lakeSoul/tool/FlinkUtil.java

+  public static CatalogTable toFlinkCatalog(TableInfo tableInfo) {
+    String tableSchema = tableInfo.getTableSchema();
+    StructType struct = (StructType) org.apache.spark.sql.types.DataType.fromJson(tableSchema);
+    Builder bd = Schema.newBuilder();


BadImport: Importing nested classes/static methods/static fields with commonly-used names can make code harder to read, because it may not be clear from the context exactly which type is being referred to. Qualifying the name with that of the containing class can make the code clearer. Here we recommend using qualified class: Schema.

Suggested change

Builder bd = Schema.newBuilder();

Schema.Builder bd = Schema.newBuilder();

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:54Z

src/main/java/org/apache/flink/lakeSoul/sink/LakSoulFileWriter.java

+          try {
+            buckets.closePartFileForBucket(BucketID);
+          } catch (Exception e) {
+            e.printStackTrace();


CatchAndPrintStackTrace: Logging or rethrowing exceptions should usually be preferred to catching and calling printStackTrace

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:55Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulBucket.java

+        case LONG_KEY_TYPE:
+          sortQueue = new PriorityQueue<>((v1, v2) -> {
+            try {
+              return (keyGen.getSimpleLongKey(v1).compareTo(keyGen.getSimpleLongKey(v2)));


UnnecessaryParentheses: These grouping parentheses are unnecessary; it is unlikely the code will be misinterpreted without them

Suggested change

return (keyGen.getSimpleLongKey(v1).compareTo(keyGen.getSimpleLongKey(v2)));

return keyGen.getSimpleLongKey(v1).compareTo(keyGen.getSimpleLongKey(v2));

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:57Z

src/main/java/org/apache/flink/lakeSoul/metaData/DataFileMetaData.java

+  }
+
+  @Override
+  public boolean equals(Object o) {


EqualsGetClass: Prefer instanceof to getClass when implementing Object#equals.

Suggested change

public boolean equals(Object o) {

if (!(o instanceof DataFileMetaData)) {

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:41:59Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/BucketStateSerializer.java

+
+
+public class BucketStateSerializer<BucketID> implements SimpleVersionedSerializer<BucketState<BucketID>> {
+  private static final int MAGIC_NUMBER = 511069049;


UnusedVariable: The field 'MAGIC_NUMBER' is never read.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:42:01Z

src/main/java/org/apache/flink/lakeSoul/sink/partition/DataPartitioner.java

+import org.apache.spark.unsafe.types.UTF8String;
+import static org.apache.spark.sql.types.DataTypes.StringType;
+
+public class DataPartitioner<String> implements Partitioner<String> {


JavaLangClash: String clashes with java.lang.String

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:42:09Z

src/main/java/org/apache/flink/lakeSoul/sink/MetaDataCommit.java

+      Path resultPath = new Path(locationPath, partition);
+      org.apache.flink.core.fs.FileStatus[] files = resultPath.getFileSystem().listStatus(resultPath);
+      DataCommitInfo dataCommitInfo =
+          partitionMetaSet(tableInfo.getTableId(), partition, partitionLists);


NULL_DEREFERENCE: object tableInfo last assigned on line 113 could be null and is dereferenced at line 136.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:42:11Z

src/main/java/org/apache/flink/lakeSoul/tool/FlinkUtil.java

+      }
+      bd = bd.column(sf.name(), tyname);
+    }
+    bd.primaryKey(Arrays.asList(hashColumn.split(",")));


NULL_DEREFERENCE: object hashColumn last assigned on line 106 could be null and is dereferenced at line 117.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:42:13Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulBucket.java

+  void sortWrite(long currentTime) throws IOException {
+    while (!sortQueue.isEmpty()) {
+      RowData poll = sortQueue.poll();
+      this.inProgressPart.write((IN) poll, currentTime);


NULL_DEREFERENCE: object LakeSoulBucket.inProgressPart last accessed on line 241 could be null and is dereferenced at line 241.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:42:15Z

src/main/java/org/apache/flink/lakeSoul/sink/bucket/LakeSoulBucket.java

+        case LONG_KEY_TYPE:
+          sortQueue = new PriorityQueue<>((v1, v2) -> {
+            try {
+              return (keyGen.getSimpleLongKey(v1).compareTo(keyGen.getSimpleLongKey(v2)));


NULL_DEREFERENCE: object returned by LakeSoulBucket.keyGen.getSimpleLongKey(v1) could be null and is dereferenced at line 109.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-07-20T07:42:16Z

src/main/java/org/apache/flink/lakeSoul/tool/LakeSoulKeyGen.java

+  }
+
+  public static int objToInt(@Nullable Object obj) {
+    return obj == null ? null : (int) obj;


NULL_DEREFERENCE: object returned by valueOf(obj.intValue()) could be null and is dereferenced at line 92.

Reply with "@sonatype-lift help" for info about LiftBot commands.
Reply with "@sonatype-lift ignore" to tell LiftBot to leave out the above finding from this PR.
Reply with "@sonatype-lift ignoreall" to tell LiftBot to leave out all the findings from this PR and from the status bar in Github.

When talking to LiftBot, you need to refresh the page to see its response. Click here to get to know more about LiftBot commands.

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sunmaosen and others added 30 commits May 23, 2022 16:28

Flink catalog management

f023f45

Flink sink writer New meta

Flink catalog management

7d4ffb7

Flink sink writer New meta

new postgres sql

0744cd5

delete

19cc9e3

Metadata for pg

666b15a

Metadata for pg

9fa84f0

Metadata for pg

e03037c

rebuild

0ee6e2a

snapshot snapshotmanagement sparkutil createtablecommand

add MetaVersion

7eb5e93

DataFileOp add new property:size and modificationTime

b1a986a

adjust metadata add datacommitinfo to metainfo

3ee6ee9

delete configuration from doMetaCommit

add new MetaCommit and DataOperate

6492299

modify data_commit_info and data_file_op:add column timestamp and cha…

4b9f78f

…nge file_exist_cols

adjust metadata for DataFileInfo

cce98c6

modify data_commit_info and data_file_op:add column timestamp and cha…

5e8f9bd

…nge file_exist_cols

change TransactionalWrite and CommitType

a555e06

re-write TransactionalCommit commit method

2500617

adjust metadata for DataFileInfo

1bc6e1e

adjust metadata for DataFileInfo

18f24e3

fix data_commit_info property bug

b70f8ec

overall code merge

76a1ed6

debug modify

ee2c0a7

debug modify:DelayedCommitProtocol.commitTask

d31dde2

debug modify: meta data format change

032925d

debug modify: data type change

3c39aef

remove materialized view related code (#32)

327cfbe

modify the type of map data in configuration transfer to jsonObject

6f8c784

DataCommitInfoDao:add detection when query criteria are empty

15d2666

adjust some bugs for mor

c8561ba

fix create table and insert suite (#25)

ce5b047

refine code

d52e180

dmetasoul01 approved these changes Jul 20, 2022

View reviewed changes

dmetasoul01 merged commit c27c933 into main Jul 20, 2022

dmetasoul01 deleted the feature_flinkOrderSink branch July 20, 2022 07:38

dmetasoul01 added this to Done in Support Flink CDC Write Jul 20, 2022

sonatype-lift bot reviewed Jul 20, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature flink order sink #56

Feature flink order sink #56

YangZH-v2 commented Jul 19, 2022 •

edited by dmetasoul01

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

sonatype-lift bot Jul 20, 2022

	String[] split = partitionKeys.split(";");
	List<String> split = Splitter.on(';').splitToList(partitionKeys);

	return keyGen.getSimpleIntKey(v1) - (keyGen.getSimpleIntKey(v2));
	return keyGen.getSimpleIntKey(v1) - keyGen.getSimpleIntKey(v2);

	public boolean equals(Object o) {
	if (!(o instanceof LakeSoulKeyAndValueSerializer)) {

	final List<Factory> factories = new LinkedList<>();
	final List<Factory> factories = new ArrayList<>();

	Builder bd = Schema.newBuilder();
	Schema.Builder bd = Schema.newBuilder();

	return (keyGen.getSimpleLongKey(v1).compareTo(keyGen.getSimpleLongKey(v2)));
	return keyGen.getSimpleLongKey(v1).compareTo(keyGen.getSimpleLongKey(v2));

	public boolean equals(Object o) {
	if (!(o instanceof DataFileMetaData)) {



		public class BucketStateSerializer<BucketID> implements SimpleVersionedSerializer<BucketState<BucketID>> {
		private static final int MAGIC_NUMBER = 511069049;

Feature flink order sink #56

Feature flink order sink #56

Conversation

YangZH-v2 commented Jul 19, 2022 • edited by dmetasoul01

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

sonatype-lift bot Jul 20, 2022

Choose a reason for hiding this comment

YangZH-v2 commented Jul 19, 2022 •

edited by dmetasoul01