Feature to return the processed keys during a data quality validation

[jmcorreia]

We have a use case, for which we want to be able to store the status of all the open and closed Data Quality problems.
For doing so, we rely on GX unexpected_index_list to have the PKs that got DQ problems and we have some custom logic on our framework to get all the unique keys on the dataset being validated - we call it processed_keys.
Afterwards we match these with the unexpected_index_list considering all history of runs, and if the previously identified bad PKs appear as good records in the following runs, we close the Data Quality problem.

This has been working until now in GX 0.18.8, even though the processing is challenging.
However, when people use row_conditions, this impose challenges as we cannot really rely on the unique keys of the Dataset, because the expectations with row_conditions are not considering the entire dataset, but a filtered version.

Of course we could try to change our logic to build the "processed_keys" for each row_condition, but this will mean additional processing and also additional data being stored. E.g. as of now we store a single processed_keys dict for the entire dataset being validated, having it 1 or 100 expectations. If we would accommodate this logic, we would have to build as much dicts as expectations with row_conditions.

IDEA PROPOSED

• as you already build the list of keys for failures, would it be possible for you to also retrieve the keys for good records or for processed records?
• or, would you be able to retrieve the keys that are ignored by applying the row_condition?
• any other idea?

Additional context

• Please let me know what you think of the idea, if you would have any other idea and if you would need more details

[adeola-ak]

what version of GX are you using?

[jmcorreia]

@adeola-ak 0.18.8 for now and in process of migrating to latest 1.x

[joshua-stauffer]

hey @jmcorreia, thanks for this question, and sorry it took a while for someone to get back to you. This is tricky because for most datasources GX never materializes the processed set. Row conditions are useful, but they do alter the actual data being processed. The expectation is effectively redefining the active batch. Better might be to allow for some sort of a query BatchDefinition, so you could move row conditions out of the expectation and into the component defining the slice of data. There's some design work to be done there, and the path forward isn't entirely clear, but I'm happy to explore that with you if that would be an improvement.