Fix #3084 - Implement missing tests #3117

pmbrull · 2022-03-03T18:12:17Z

Describe your changes :

Fixes #3084, among other small tunings. I'll open new issues for the rest of the validations, but this was getting big already.

Implemented columnCount metric and its validation, among other straightforward validations.
Updated the type on the JSON Schema for ColumnValuesToBeNotInSet. The idea is that we need a list of whatever, not a Dict.
Now we are passing session and the ORM table to the validate call. This is interesting as it allows validation implementations to prepare and run their own metrics. This is required for metrics that have input parameters, such as a set of values for ColumnValuesToBeNotInSet. In this scenario, we cannot reuse any profiler metric, but rather we need some one-shot specific implementation for each column the test is configured. It is a small change and opens the door for more complicated testing
Fixed the concat function implementation for Redshift
Added a better exception and rollback handling on errors computing metrics.
Deleted the gitbook publish CI

Thanks, happy to discuss 🙏

Type of change :

Frontend Preview (Screenshots) :

For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

I have read the CONTRIBUTING document.
I have performed a self-review of my own.
I have tagged my reviewers below.
I have commented on my code, particularly in hard-to-understand areas.
My changes generate no new warnings.
I have added tests that prove my fix is effective or that my feature works.
All new and existing tests passed.

Reviewers

github-actions · 2022-03-03T18:12:46Z

Schema Change Detected. Needs ingestion-core version bump

Please run make core_bump_version_dev in the project's root and commit the changes to _version.py in this PR. Please ignore if this has been handled already.