-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Transform] data quality for null data rate #1978
Conversation
Please solve compile problem first |
The error is ‘Can't use 'tar -xzf'’ ,so how to solve |
I rerun it, seem like already pass |
docs/en/transform/nullRate.md
Outdated
@@ -0,0 +1,67 @@ | |||
# EncryptedPhone |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The header wrong
docs/en/transform/nullRate.md
Outdated
|
||
|
||
|
||
### field [string] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be string_list
docs/en/transform/nullRate.md
Outdated
|
||
Which fields do you want to monitor . | ||
|
||
### rates [double] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be double_list
object NullRateConfig { | ||
val FIELDS = "fields" | ||
val RATES = "rates" | ||
val IS_THROWEXCEPTION = "throwException_enable" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
throwException_enable
should be throw_exception_enable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done all
.add("setting_rate", DataTypes.LongType) | ||
.add("null_rate", DataTypes.LongType) | ||
.add("rate_percent", DataTypes.LongType) | ||
env.getSparkSession.createDataset(nullRateRows)(RowEncoder(schema)).createOrReplaceTempView(config.getString(Plugin.RESULT_TABLE_NAME)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The RESULT_TABLE_NAME
shouldn't use in here, SeaTunnel will auto create temp view for Dataset which return by this method, and named RESULT_TABLE_NAME
, so other plugin can drectly use it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Purpose of this pull request
#1977(#1977)
Check list
New License Guide