1. Introduction
1.1 Context
With the current structure of the data quality, TestSuite are used as a logic unit to schedule and execute test cases. This structure forces a user to define a schedule for tests across multiple table that can have different refresh logic. To accomodate the DQ requirements, the same tests may need to be created in different TestSuite.
To prevent unnecessary duplication of tests and allow the tests execution to be linked to the table update schedule, while keeping the flexibility provided by Test Suite, the team will move the execution logic of testCase at the table level while allowing users to logically group testCase into testSuite for reporting (visualization, alerting, etc.)
2. Product and Technical Requirements
2.1. User Stories
2.1.1. Use case 1: As an owner of the table I want to ensure data quality
As an owner, I have written a bunch of tests for the table. I schedule these tests to be run periodically and publish the test results to the table. When there are test failures, I look into the issues and fix them or mark them as non-issues. The published test results also show my data consumers the current quality of data and how it has changed over time.
2.1.2. Use case 2: As a data consumer, I want to make sure my upstream data quality is good
I use data to produce reports and dashboards. I use multiple tables and columns from them to produce the data assets. The quality of the data asset I produce depends on the upstream data that I use. To do this, I create a Test Suite (or any other abstraction) that groups all the tests to ensure the quality of upstream data I use. I pick table tests or column tests from upstream tables and add them to my test suite. I can see the results of the tests by going to the test suite and ensuring that there are no data quality issues.
2.1.3. Use case 3: As a data consumer, I want to add tests to a table
I want to add tests to a table because I am making certain assumptions about the data (cardinality for example, column removal), and when something breaks that assumption, I want to know so I can change my pipelines to handle it.
2.2. Technical Requirements
2.2.1. Move logical execution of tests to the table level [UI/ingestion]
The ability to create test suite as an execution unit should be removed. The workflow should only allow to add test cases and the test suite creation step should be removed. A default test suite with the table fqn will be used logically to group test cases. A type parameter of value SYSTEM should be passed to signify this test suite is a system created tests suite
2.2.2. Ingestion workflow update [Ingestion]
The execution logic of test suites in the ingestion framework should be updated to take into account the new execution approach. Importantly, only system test suites should be executed.
Work for defining the CLI logic update needs to be done as well. We will need to:
- only allow execution of system test suites
- check that all test cases defined in the test suite relate to the same table
2.2.3. Handle migrations [backend]
- Move the existing test suite from logic execution units to logic grouping units
- Move test cases from existing test suites to table-level test suites
2.2.4 Update table API endpoint for test case results [backend]
Similar to profiler data, allow test case results to be fetch at the table level
2.2.5. Implement search logic for test case/suites [backend]
With the improvements related to the UI, we want to make test suite/case searchable. This will require the implementation of the search functionality and the indexation of test cases
2.2.6. Bulk add test cases to tests suites [backend]
2.2.7. UI improvements [UI]
On the Quality tab 3 views should be offered -- see below for the wireframes and details.
By Test Suite
This view should only display the test suite with type USER. These test suites will be test suites defined by users that logically group test cases. Functionally, the flow will be similar to the existing test suite flow with the following differences:
- instead of creating new test cases users will choose existing test cases to add to a test suite
- there will be no scheduling
On this view users should be allowed to create new test suites. Clicking on a test suite should redirect to the test case level view where only test case for the particular test suite will be shown
By Table
The table view will display a summary of the data quality at the table view. Clicking on a specific table should redirect to the table -> data quality page.
By Test Case
When navigating to the test case view all the test cases should be displayed.
General remarks
- Tests should be sorted by status (failed first) and execution date (most recent)
- The "last results" squares should display the total number of test failures aggregated at the day (for the last 7 days).
- "Last run" should represent the most recent execution (for tests suite view, it should show the most recent execution of a test case in the test suite)
2.2.9. Add a status button [UI/Backend/Ingestion]
Users should be able to visualize the status of a failed test case (i.e. New, Ack, Resolved). If a test fails, and no status was set on the previous run then set the status to New. If a test fails and the status on the previous run was Ack, then reuse this status. If the test pass set no status.
From the UI standpoint, a user should be able to mark the test case as Ack or Resolved.
2.3. Future Goals
- Permission at the test suite level (e.g. only show my test suite to my team)
- Rules to define what tests should be considered as pertinent to decide whether my table is healthy or not (i.e. technical vs business tests)
- Allow users to set alerts when creating a new test suite
3. Work
3.1. Prioritization
Design
Frontend
Ingestion
Backend
1. Introduction
1.1 Context
With the current structure of the data quality, TestSuite are used as a logic unit to schedule and execute test cases. This structure forces a user to define a schedule for tests across multiple table that can have different refresh logic. To accomodate the DQ requirements, the same tests may need to be created in different TestSuite.
To prevent unnecessary duplication of tests and allow the tests execution to be linked to the table update schedule, while keeping the flexibility provided by Test Suite, the team will move the execution logic of testCase at the table level while allowing users to logically group testCase into testSuite for reporting (visualization, alerting, etc.)
2. Product and Technical Requirements
2.1. User Stories
2.1.1. Use case 1: As an owner of the table I want to ensure data quality
As an owner, I have written a bunch of tests for the table. I schedule these tests to be run periodically and publish the test results to the table. When there are test failures, I look into the issues and fix them or mark them as non-issues. The published test results also show my data consumers the current quality of data and how it has changed over time.
2.1.2. Use case 2: As a data consumer, I want to make sure my upstream data quality is good
I use data to produce reports and dashboards. I use multiple tables and columns from them to produce the data assets. The quality of the data asset I produce depends on the upstream data that I use. To do this, I create a Test Suite (or any other abstraction) that groups all the tests to ensure the quality of upstream data I use. I pick table tests or column tests from upstream tables and add them to my test suite. I can see the results of the tests by going to the test suite and ensuring that there are no data quality issues.
2.1.3. Use case 3: As a data consumer, I want to add tests to a table
I want to add tests to a table because I am making certain assumptions about the data (cardinality for example, column removal), and when something breaks that assumption, I want to know so I can change my pipelines to handle it.
2.2. Technical Requirements
2.2.1. Move logical execution of tests to the table level [UI/ingestion]
The ability to create test suite as an execution unit should be removed. The workflow should only allow to add test cases and the test suite creation step should be removed. A default test suite with the table fqn will be used logically to group test cases. A
typeparameter of valueSYSTEMshould be passed to signify this test suite is a system created tests suite2.2.2. Ingestion workflow update [Ingestion]
The execution logic of test suites in the ingestion framework should be updated to take into account the new execution approach. Importantly, only system test suites should be executed.
Work for defining the CLI logic update needs to be done as well. We will need to:
2.2.3. Handle migrations [backend]
2.2.4 Update table API endpoint for test case results [backend]
Similar to profiler data, allow test case results to be fetch at the table level
2.2.5. Implement search logic for test case/suites [backend]
With the improvements related to the UI, we want to make test suite/case searchable. This will require the implementation of the search functionality and the indexation of test cases
2.2.6. Bulk add test cases to tests suites [backend]
2.2.7. UI improvements [UI]
On the
Qualitytab 3 views should be offered -- see below for the wireframes and details.By Test Suite
This view should only display the test suite with
typeUSER. These test suites will be test suites defined by users that logically group test cases. Functionally, the flow will be similar to the existing test suite flow with the following differences:On this view users should be allowed to create new test suites. Clicking on a test suite should redirect to the test case level view where only test case for the particular test suite will be shown
By Table
The table view will display a summary of the data quality at the table view. Clicking on a specific table should redirect to the table -> data quality page.
By Test Case
When navigating to the test case view all the test cases should be displayed.
General remarks
2.2.9. Add a status button [UI/Backend/Ingestion]
Users should be able to visualize the status of a failed test case (i.e.
New,Ack,Resolved). If a test fails, and no status was set on the previous run then set the status toNew. If a test fails and the status on the previous run wasAck, then reuse this status. If the test pass set no status.From the UI standpoint, a user should be able to mark the test case as
AckorResolved.2.3. Future Goals
3. Work
3.1. Prioritization
Design
Frontend
Ingestion
Backend
Bulk Add test cases [Backend]