Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix #3248 & #3251 - Update metrics and column profile #3262

Merged
merged 11 commits into from Mar 8, 2022
Merged

Fix #3248 & #3251 - Update metrics and column profile #3262

merged 11 commits into from Mar 8, 2022

Conversation

pmbrull
Copy link
Collaborator

@pmbrull pmbrull commented Mar 8, 2022 •
edited

Describe your changes :

This PR fixes #3248 and fixes #3251.

We have:

  • Removed missingPercentage and missingCount from the columnProfile. It just makes sense in tests.
  • Added distinctProportion to columnProfile
  • Refactored and properly differentiated unique and distinct metrics
  • Updated metrics test with more data
  • Updated profiler QueryMetric runner
  • Added missing @property to a bunch of metrics.

Thanks

Type of change :

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Frontend Preview (Screenshots) :

For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

  • I have read the CONTRIBUTING document.
  • I have performed a self-review of my own.
  • I have tagged my reviewers below.
  • I have commented on my code, particularly in hard-to-understand areas.
  • My changes generate no new warnings.
  • I have added tests that prove my fix is effective or that my feature works.
  • All new and existing tests passed.

Reviewers

@github-actions
Copy link

github-actions bot commented Mar 8, 2022

Schema Change Detected. Needs ingestion-core version bump

Please run make core_bump_version_dev in the project's root and commit the changes to _version.py in this PR. Please ignore if this has been handled already.

Sachin-chaurasiya
Sachin-chaurasiya requested changes Mar 8, 2022
View reviewed changes
Copy link
Member

@Sachin-chaurasiya Sachin-chaurasiya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments

@pmbrull
Copy link
Collaborator Author

pmbrull commented Mar 8, 2022

Few comments

Thanks, @Sachin-chaurasiya. Pushed the changes 🙏

@sonarcloud
Copy link

sonarcloud bot commented Mar 8, 2022

[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@sonarcloud
Copy link

sonarcloud bot commented Mar 8, 2022

[catalog] Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

92.9% 92.9% Coverage
0.0% 0.0% Duplication

@pmbrull pmbrull merged commit 94d7500 into open-metadata:main Mar 8, 2022
@pmbrull pmbrull deleted the issue-3248 branch March 8, 2022 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Sachin-chaurasiya Sachin-chaurasiya Sachin-chaurasiya approved these changes

@harshach harshach Awaiting requested review from harshach

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Data Quality - Missing Count & Percentage Data Quality - Review unique vs distinct
2 participants
@pmbrull @Sachin-chaurasiya