You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Here is a table with columns(A B C D ) , Column A is primary key , All fields are type string.
Spark reads a batch of json data from kafka and converts to Dataframe to upsert. But Dataframe may contains "null" or null for some cells because of some missing fields in kafka(.e.g {A:A1,B:B4,C:C4,D:D4}{A:A2,C:C5}{A:A3,B:B5,D:D5}) which could lead to unexpected result in new table.
Since LakeSoul has MergeOperator feature, we could create a new mergeoprator to deal with this case by ignoring null values
The text was updated successfully, but these errors were encountered:
dmetasoul01
changed the title
New Feature:New MergeOprator for upsert in some scenarios
New Feature:New MergeOperator for upsert to ignore null fields by default
May 26, 2022
Here is a table with columns(A B C D ) , Column A is primary key , All fields are type string.

Spark reads a batch of json data from kafka and converts to Dataframe to upsert. But Dataframe may contains "null" or null for some cells because of some missing fields in kafka(.e.g {A:A1,B:B4,C:C4,D:D4}{A:A2,C:C5}{A:A3,B:B5,D:D5}) which could lead to unexpected result in new table.
Since LakeSoul has MergeOperator feature, we could create a new mergeoprator to deal with this case by ignoring null values
The text was updated successfully, but these errors were encountered: