-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Smooth Validation and Profiling Performance #619
Conversation
action before refactor, with associated tests.
66bff11
to
f103be9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
return column.withColumn('__success', success_udf(column[0])) | ||
if None in value_set: | ||
# spark isin returns None when any value is compared to None | ||
logger.error("expect_column_values_to_be_in_set cannot support a None in the value_set in spark") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
such nice error messages
{'expectation_type': 'expect_column_values_to_be_in_set', | ||
'kwargs': {'column': 'D', 'value_set': ['e', 'f', 'g', 'h']}} | ||
] | ||
|
||
sub1 = df[:3] | ||
|
||
sub1.discard_failing_expectations() | ||
self.assertEqual(sub1.find_expectations(), exp1) | ||
# PY2 sorting is allowed and order not guaranteed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feels like sorting is a function that ExpectationSuite.get_config
(or similar) should eventually handle.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
This PR addresses issues related to performance and ease of profiling larger datasets.
In addition, this PR changes validation by default to detect column-type expectations and group them. This is nearly zero-cost in all circumstances and can speed up cases where iterating over the same column has a cache benefit. It also demonstrates how using a ValidationOperator may affect validation order or other characteristics.