Fix #2981 - Update Profile to match TableProfile

[pmbrull]

Describe your changes :

This PR fixes #2981 and closes #2980

Reviewing the API definitions, we noticed that our current approach to computing profiling data was not properly synced with the TableProfile definition.

Instead of adding yet-another-step and mapping the results artificially, we have done a refactor on how the profiling data gets computed. We are now relying on a single Profiler object, which is able to compute both table and column metrics and itself handles all the columns in the table.

Its results can then be safely parsed into TableProfile without any issues.

In terms of the JSON Schema, we modified a bit the types of the metrics to better represent the results and added a couple of metrics that were not present and we are already computing. We also changed a few names of the metrics. ⚠️ This might have an impact on the UI as well @Sachin-chaurasiya, @darth-coder00 ⚠️ However, we still need to work on the UI anyway for this EPIC so we could maybe live with it for a while.

We also added a method to manually name the metrics, which will make it easier to map with TableProfile specs.

An interesting addition is the function add_props in metrics/core. As we now perform the metrics instantiation inside the Profiler, we are passing the class definition as an input parameter. However, there are metrics that require specific attributes such as bins or expression. To have the Profiler as generic as possible and not keep checking specific metrics for individualities, add_props dynamically prepares a new class definition overriding the __init__ method to add any required attribute we pass as **kwargs.

This means that we can run new_hist = add_props(bins=5)(Metrics.HISTOGRAM.value) and new_hist will be a completely independent and new class definition that replicates Metrics.HISTOGRAM.value but will assign bins=5 to all its instantiations.

Happy to discuss!

Thanks

Type of change :

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Frontend Preview (Screenshots) :

For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

• I have read the CONTRIBUTING document.
• I have performed a self-review of my own.
• I have tagged my reviewers below.
• I have commented on my code, particularly in hard-to-understand areas.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• All new and existing tests passed.

Reviewers

[github-actions]

Schema Change Detected. Needs ingestion-core version bump

Please run make core_bump_version_dev in the project's root and commit the changes to _version.py in this PR. Please ignore if this has been handled already.

[pmbrull]

If these changes are an issue now, I can revert this and change the naming on the python side. We can tackle it later if needed

[harshach]

lets not change it. UI can fix it.

[pmbrull]

Ill revert the datatype changes and handle them separatedly as it is a handful

[harshach]

@pmbrull overall LGTM. Left couple of comments. If we don't need numpy lets not include it

[harshach]

also we need to update the sample data

[sonarcloud]

[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[pmbrull]

Hi @harshach, reverted all other changes and removed numpy dependency. The only change I'd like to stick here is: adding minLength and maxLength and updating the type for min and max. Looks like the tests are passing and this way I can compute it for numbers and strings

[sonarcloud]

[catalog] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication