Table checks frontend component

[danwom]

Adds an RFC to propose:

-a frontend component for displaying 3-valued table status: number of checks, number passing, and an optional timestamp
-stubbing of the backend for this component

[feng-tao]

lgtm

[mgorsk1]

when we are talking about 3 of 10 checks passed which one do we mean:

• out of last 10 unit tests 3 of them failed
  or
• the last unit test on this table consisted of 10 checks and 3 of them failed?

I am pretty sure you mean the latter one but wanted to be 200% sure.

[danwom]

The way I interpret it is: Checks run on various cadences. Some tests run hourly, some might run daily or even weekly. For each individual check, how many of the last run passed or failed.

This interpretation is left up to the specific implementation. We define an API with fields like num_checks_passed, num_checks_total, and last_run_timestamp. These can mean slightly different things if you wish.

[mgorsk1]

Do you already have in mind some tools from which these quality checks could have been collected? I can think of Great Expectations and dbt tests but would be useful to have broader list. It's also something @sewardgw had good point about the interface to be generic enough we are not forced to do backwards incompatible changes later on.

From what I can see the mockups you introduced somewhat adhere to GE format but I am not sure about dbt (cause I don't have exp with it).

[dkunitsk]

We are not building this with a particular tool in mind. At the 1M-mile view, we think our proposed definition is so generic that it could crudely fit anything. The idea is to start here and expand once people start plugging in particular systems.

At Lyft we do have a DQ system we're planning to plug in, but it exposes a much more complicated API and semantics which we are not taking into account here. Instead, we asked ourselves what would be the most generic first step.

[dkunitsk]

@mgorsk1 @sewardgw Thanks for your thoughts above. Your question around comparing the API vs offline databuilder approaches makes a lot of sense.

At this time, we are not ready to add full first class data quality checks to Amundsen. We aren't even ready to put forward a good generic definition of a check, as it might have a lot of dimensions and we're not ready to pin it down (condition, time range, columns validated, correctness/failure percentage, severity, owner, etc). This would be a great direction, but not what we were planning to tackle.

Instead, we're proposing a simple "Table Checks" frontend component to display limited pass/fail state. Originally, we were going to power this component with a customizable client (as this follows some prior art in Amundsen) but given your feedback we have an alternate suggestion:

Add a placeholder endpoint (/api/quality/v0) in the frontend service which returns a dummy response (e.g. "0 out of 0 checks successful"). Companies who have an API to hook up to this frontend component could override the endpoint implementation with a call to their API. Otherwise, this state would be a clean starting point for adding backend ingestion down the line.

This has the advantage of moving in the general correct direction (a pluggable "checks" frontend component) without making premature decisions, since we'd be starting with the the most bare-bones definition possible.

[mgorsk1]

What I am missing about this proposal is some example/guidance on what DQ systems you have in mind to be used with current approach @dkunitsk Is it something custom you have in Lyft? Having such list enables us to expand Future possibilities section with more guided way to build on top of this.

It is also important to note that quality checks might be on column level, not table level and most probably these are even more important. For example Great Expectation tests are defined both on table (I expect this table to have at least N rows) and column level (I expect this column to have those 5 unique values only) separately.

I agree with @sewardgw on SLA and push/pull semantics and that either way it's possible to have data in Amundsen as soon as it's possible.

[dkunitsk]

@mgorsk1 unfortunately we don't have most of the answers at this time. You're very right that a big portion of data quality will be at the column level, and we will be working in that direction in the near future.

At the moment we're proposing only:

• a frontend component for displaying 3-valued table status: number of checks, number passing, and an optional timestamp
• stubbing of the backend for this component

In some sense, this is un-opinionated enough that one could crudely plug in any DQ system (even column-level systems). And we believe it's a generic starting point for all generalizations discussed - databuilders, push/pull questions, check definition, etc.

Given the good feedback above, we have a much better grasp of what the community is looking for in a generic DQ check feature and we'll be tailoring future proposals according to this feedback.

In the meantime, we'll adjust this RFC to make our scope more clear so we don't get stuck in the deeper questions just yet.

[dkunitsk]

Note: this is not so much an alternative but rather an addition.

[dkunitsk]

+1. After the pared down scope, this leaves room for multiple directions on the unsolved questions (push/pull, specific integrations, etc).