(enhancement): Extend get and update ruleset DQ methods #1882
Conversation
jaidisido
requested review from
cnfait,
kukushking,
LeonLuttenberger and
malachi-constant
| @@ -134,6 +136,8 @@ def update_ruleset( | |||
| Ruleset name. | |||
| updated_name : str | |||
| New ruleset name if renaming an existing ruleset. | |||
| mode : str | |||
| overwrite (default) or upsert. | |||
There was a problem hiding this comment.
overwrite and upsert are the only modes I can think of
| if mode not in ["overwrite", "upsert"]: | ||
| raise exceptions.InvalidArgumentValue("`mode` must be one of 'overwrite' or 'upsert'.") | ||
|
|
||
| if mode == "upsert": |
There was a problem hiding this comment.
Could not find a better way to do an upsert in pandas
| for ruleset_name in ruleset_names: | ||
| rules = cast(str, _get_ruleset(ruleset_name=ruleset_name, boto3_session=boto3_session)["Ruleset"]) | ||
| df = _rules_to_df(rules=rules) | ||
| if len(ruleset_names) > 1: |
There was a problem hiding this comment.
I think a column with the ruleset name should only be added if there is multiple ones. But let me know if you disagree
There was a problem hiding this comment.
What's the major downside to adding the ruleset name always? I like the idea of the structure of the DataFrame to be consistent (i.e. it should either always have ruleset).
| for ruleset_name in ruleset_names: | ||
| rules = cast(str, _get_ruleset(ruleset_name=ruleset_name, boto3_session=boto3_session)["Ruleset"]) | ||
| df = _rules_to_df(rules=rules) | ||
| if len(ruleset_names) > 1: |
There was a problem hiding this comment.
What's the major downside to adding the ruleset name always? I like the idea of the structure of the DataFrame to be consistent (i.e. it should either always have ruleset).
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Feature or Bugfix
Detail
get_rulesetshould accept multiple rulesets and combine them to a data frameupdate_rulesetjust overwrites the existing ruleset. Introducing amodeargument (overwrite, upsert) to handle upsertstry_itto handle request throttlingRelates
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.