You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Interest in a constraint-oriented SQL Expectation pattern built on UnexpectedRowsExpectation?
#12088
I've been working with UnexpectedRowsExpectation for logical and cross-field validation rules and have found it to be a very flexible escape hatch for validations that are naturally expressed in SQL. This may be relevant to additional SQL support question here
One pattern I keep running into, though, is that many rules are easier to author and reason about as constraints that should be true, rather than as queries that return violating rows.
For example, a rule might naturally be expressed as:
age >=18OR age IS NULL
whereas UnexpectedRowsExpectation requires expressing the inverse:
SELECT*FROM {batch}
WHERE NOT (age >=18OR age IS NULL)
For simple cases this is trivial, but it becomes less intuitive for more complex logical rules, especially when expectations are serialized or generated from another declarative representation.
I've been experimenting with constraint-oriented expectations such as:
ExpectRowsToSatisfy(
condition="age >= 18 OR age IS NULL"
)
Internally, these could still use the existing UnexpectedRowsExpectation / query metric machinery by compiling the constraint into a query that returns the violating rows.
This seems like it could provide a useful middle ground between:
the built-in, purpose-specific GX Expectations; and
dropping all the way down to an arbitrary UnexpectedRowsExpectation.
I'm curious how the GX maintainers/community think about this pattern.
Would there be interest in any of the following?
A documented recipe for building constraint-oriented Expectations on top of UnexpectedRowsExpectation.
A reusable base class/helper that accepts a SQL predicate and handles inversion/query construction.
A small set of generally useful constraint-style Expectations, such as ExpectColumnToBeNullWhere, ExpectColumnToEqualWhere, or a general ExpectRowsToSatisfy.
Keeping this entirely in user-defined Expectations, but providing guidance on the preferred architecture for implementing them.
One thing I particularly like about this approach is that the user-facing rule can remain declarative and close to the business constraint, while GX can continue to use unexpected rows as the standard diagnostic/result representation.
I'd be interested in contributing a recipe or implementation if this fits the direction GX wants to take.
Does this seem like a useful abstraction, or is UnexpectedRowsExpectation intentionally meant to remain the primary interface for this class of validation?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I've been working with
UnexpectedRowsExpectationfor logical and cross-field validation rules and have found it to be a very flexible escape hatch for validations that are naturally expressed in SQL. This may be relevant to additional SQL support question hereOne pattern I keep running into, though, is that many rules are easier to author and reason about as constraints that should be true, rather than as queries that return violating rows.
For example, a rule might naturally be expressed as:
whereas
UnexpectedRowsExpectationrequires expressing the inverse:For simple cases this is trivial, but it becomes less intuitive for more complex logical rules, especially when expectations are serialized or generated from another declarative representation.
I've been experimenting with constraint-oriented expectations such as:
or more generally something like:
Internally, these could still use the existing
UnexpectedRowsExpectation/ query metric machinery by compiling the constraint into a query that returns the violating rows.This seems like it could provide a useful middle ground between:
UnexpectedRowsExpectation.I'm curious how the GX maintainers/community think about this pattern.
Would there be interest in any of the following?
UnexpectedRowsExpectation.ExpectColumnToBeNullWhere,ExpectColumnToEqualWhere, or a generalExpectRowsToSatisfy.One thing I particularly like about this approach is that the user-facing rule can remain declarative and close to the business constraint, while GX can continue to use unexpected rows as the standard diagnostic/result representation.
I'd be interested in contributing a recipe or implementation if this fits the direction GX wants to take.
Does this seem like a useful abstraction, or is
UnexpectedRowsExpectationintentionally meant to remain the primary interface for this class of validation?All reactions