New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add strict mode in Routine load, Stream load and Mini load #1677
Conversation
gensrc/thrift/FrontendService.thrift
Outdated
@@ -522,7 +522,8 @@ struct TStreamLoadPutRequest { | |||
15: optional string partitions | |||
16: optional i64 auth_code | |||
17: optional bool negative | |||
18: optional i32 timeout | |||
18: optional bool strictMode | |||
19: optional i32 timeout |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't modify the index of old field. If you change the index, this may not compatible with old format
@@ -1160,6 +1169,10 @@ public void readFields(DataInput in) throws IOException { | |||
|
|||
origStmt = Text.readString(in); | |||
|
|||
if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_59) { | |||
strictMode = in.readBoolean(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you make all old routine load job to strict mode. This may cause many data to be filtered
@@ -57,6 +57,7 @@ | |||
private String partitions; | |||
private String path; | |||
private boolean negative; | |||
private boolean strictMode = RoutineLoadJob.DEFAULT_STRICT_MODE; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why put this DEFAULT_STRICT_MODE in RoutineLoadJob?
@@ -105,6 +105,7 @@ | |||
.add(MAX_BATCH_INTERVAL_SEC_PROPERTY) | |||
.add(MAX_BATCH_ROWS_PROPERTY) | |||
.add(MAX_BATCH_SIZE_PROPERTY) | |||
.add(LoadStmt.STRICT_MODE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we update our Document about this property?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
1c370a1
to
239467e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
No description provided.