Fix #2875 - Profiler API Sink #3011
Conversation
# Conflicts: # ingestion-core/src/metadata/_version.py
This reverts commit 50702c3.
|
|
||
| if record.record_tests: | ||
| for table_test in record.record_tests.table_tests: | ||
| self.metadata.add_table_test( |
There was a problem hiding this comment.
Update test definitions and add testCaseResult info
| logger = logger() | ||
|
|
||
|
|
||
| def column_values_to_be_between( |
There was a problem hiding this comment.
Sample test case implementation
| """ | ||
| The required Metric is not available in the profiler results | ||
| @singledispatch | ||
| def validate(test_case, *args, **kwargs) -> TestCaseResult: |
There was a problem hiding this comment.
Generic function to handle all test implementations
|
Opened #3012 to handle py-tests issue from the last few PRs. Some dependency updates broke the |
|
For some reason, a class that gets generated for me as |
# Conflicts: # ingestion/tests/integration/orm_profiler/test_orm_profiler.py
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
|
[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed! |
Describe your changes :
This PR fixes #2875
We have removed the deprecated grammar approach to validation definitions and instead, use pydantic models coming from the API. These changes can be seen in
validations/core.py.The ORM Processor now handles these typed tests and returns
TestCaseResultto be updated in the Sink.Regarding test definition, we support both adding tests directly as part of the workflow config or externally via the API. In both cases, we handle the tests for each record and run them. If the tests are added in the workflow, they will be PUT in the sink as well to update the Table Entity.
To handle test APIs, we have updated the OMeta Python API with the required methods
Type of change :
Frontend Preview (Screenshots) :
For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.
Checklist:
Reviewers