Add an optional pre-run check to Checkpoints

[GilShaulShay]

Hi! I’m building a prototype that runs the same GX checks against a database under normal and disrupted conditions.
One situation I’m testing is validation starting while a data load is still running. GX correctly reports missing rows, but I’d like to distinguish an incomplete load from a completed dataset that fails validation.

I looked at Checkpoint Actions, but those run after validation. Is there a supported way to plug in a check before validation starts?
If not, would you consider an optional pre-validation hook? Users would provide a Python function that checks readiness, for example by querying a load-tracking table or calling an upstream API. For a load-complete marker, the check would need to confirm it belongs to the batch being validated, not a previous load.

If the function reports “not ready” or cannot determine readiness, GX wouldn’t start validation. It would return a “not run”(or aborted) outcome with the reason, so the pipeline could decide whether to retry later or stop. Otherwise, the same GX checks would run unchanged, and any data-quality failures would still be reported as failures.

I can put this logic before checkpoint.run() in my own code. What I’m suggesting is a standard place to attach it and a standard way to report that validation didn’t run, without treating it as a pass or a failure.

[joshua-stauffer]

Hi @GilShaulShay, thanks for the question, this is an interesting use case.

Is there a supported way to plug in a check before validation starts?

There isn't, and my instinct that your orchestrator is likely better suited to perform pre-validation readiness checks than GX is, particularly if you're calling an API. Beyond the general question of library scope, Checkpoints need to be serializable, so I'm not sure it's feasible to add a pre-validation hook as a Python callable.

On the other hand, if you're checking state of a load-tracking table, we could reframe this as a data quality question. The pattern would be to set up a table asset and batch definition for your load-tracking table using the same grain partitioner that you use for the primary Checkpoint (e.g. add_batch_definition_daily), and validate it against an Expectation with a row condition that filters down to whatever signal the table uses to indicate that the batch is fully loaded, for instance ExpectTableRowCountToEqual(value=1, row_condition=Column("status") == "complete"). Then the primary Checkpoint can sit downstream of that validation, and failure on that validation is the "not ready" signal you're looking for.

Does that work for your use case? If not, I'm curious to better understand why GX would be better suited to own the pre-flight checks than your orchestrator pipeline. Hope that helps!

[GilShaulShay]

Thanks, that helps. Checking the load-tracking table with GX makes sense for this case, and I see your point about serialization.
I’ll try a separate readiness validation and let the pipeline decide whether to run the main suite. That would keep the data checks unchanged.
One thing I’d like to keep clear in the reporting is that a failed readiness check means the main suite was not run, rather than that its checks failed. Is there an existing GX pattern for representing that, or would you keep that reporting in the pipeline too?