Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bugfix - Data Quality #269

Merged
merged 27 commits into from
Jan 22, 2024
Merged

Bugfix - Data Quality #269

merged 27 commits into from
Jan 22, 2024

Conversation

jwaspin
Copy link
Contributor

@jwaspin jwaspin commented Dec 20, 2023 •
edited
Loading

Closes #223
Closes #224
Closes #225
Closes #226
Closes #227
Closes #228
Closes #229
Closes #230
Closes #255
Closes #256
Closes #262

Changes

  • Incremented patch version
    • package.json
    • schema/schema.json
  • Updated .gitignore
  • Fixed Data Quality schemas that were not correct and added missing pieces
    • Fixed the ordering of the components
    • Evaluation Method
      • object not array
      • changed dateTime to an array to handle 2 dateTime values (start & end)
    • Coverage Result
      • Modified resultContent to be resultContentDescription which is a CoverageDescription object
      • added new qualityResultFile schema
      • changed resultFile to be the qualityResultFile
      • resourceFormat changed to the format object
  • Added required fields / rules
  • Added descriptions
  • Range element description schema cleaned up

Testing

This version of the schema can be inspected with the schema viewer at the url below

http://23.23.4.125:8051/ (current version is 2.8.2-beta.7)

@jwaspin jwaspin marked this pull request as draft December 20, 2023 14:09
@jwaspin jwaspin mentioned this pull request Dec 20, 2023
Closed
8 tasks
@jwaspin jwaspin changed the title Bugfix - Data Quality Evaluation Method Bugfix - Data Quality Dec 20, 2023
jwaspin and others added 16 commits December 20, 2023 12:20
@jwaspin
Fix Required List
08af473
@jwaspin
Fix Report Schema
e9abcef
@jwaspin
Merge branch 'dev' into bugfix/data-quality-evaluation-method
052bef9
@jwaspin
Add Additional Enum Options
cabadb7
@jwaspin
Add Required
94bb3cd
@jwaspin
Fix Evaluation Method
6dbcc52 
Changed dateTime to an array.
Added enum options for type.
@jwaspin
Add Required
3452282
@jwaspin
Add Required Fields
8ca2849
@jwaspin
Add Descriptions
4c906fa
@J-Oliveros
add maxItem rule to schema
fee3ca6
@jwaspin
Update standaloneQualityReport.json
3625793
@jwaspin
Updates
e1fad0b 
Version numbers.
Requirements.
Add derivedElementReport.
Cleanup rangeElementDescription.
@jwaspin
Merge branch 'bugfix/data-quality-evaluation-method' of github.com:ad…
2434581 
…iwg/mdJson-schemas into bugfix/data-quality-evaluation-method
@jwaspin
Update dataQuality.json
ffa3136 
Remove max items. It does not appear to do anything.
@jwaspin
Remove beta from version
7369cdf
@jwaspin
Merge branch 'bugfix/data-quality-evaluation-method' of github.com:ad…
a3a21f2 
…iwg/mdJson-schemas into bugfix/data-quality-evaluation-method
@jwaspin jwaspin marked this pull request as ready for review January 11, 2024 18:55
@jwaspin
Copy link
Contributor Author

jwaspin commented Jan 11, 2024

@dwalt & @hmaier-fws This PR is ready for review, but I'm not sure it 100% closes all those issues. I wasn't sure about a couple things like if all the scopes were required or just the top level one for data quality. Any small things I can fix immediately, but if we need some discussion on anything should we create new issues and close these ones?

One specific part to be aware of is the coverageResult class is listed in this PR, we might want to pull it out but there is an implementation for it so it might be better to create new issues to fix anything after we discuss the expected implementation (that's the one we were discussing the other day that needs to be reviewed).

@jwaspin jwaspin requested a review from dwalt January 17, 2024 02:22
@jwaspin
Fix Coverage Result
16958a1 
Add qualityResultFile.
Update required fields.
@jwaspin
Fix Coverage Result
5d5d223 
resultContent is actually resultContentDescription and is a Coverage Description object not an array.
@jwaspin
Fix Required Fields
0ced67a
@jwaspin
Copy link
Contributor Author

jwaspin commented Jan 18, 2024

Note that coverageResult has been updated and should be completed at this point. I did have to change some things based on what the schema looked like it should have been, those changes are noted in my comment followed by the modified description (which I've done in all the issues related to a new class if I had to change anything).

All of data quality should be done, so there are no known issues at this point.

@jwaspin
Copy link
Contributor Author

jwaspin commented Jan 18, 2024

In this issue: #262

It mentions "Translator UI not working, network error" but I'm not sure what this refers to. If I were to guess, this is actually an issue in mdTools, not mdJson-schemas, because that translator UI is not working, but there's already an issue for that in the mdTools repository.

hmaier-fws
hmaier-fws approved these changes Jan 22, 2024
View reviewed changes
Copy link
Contributor

@hmaier-fws hmaier-fws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finally getting to this....

@hmaier-fws hmaier-fws merged commit b88a83d into dev Jan 22, 2024
@jwaspin jwaspin deleted the bugfix/data-quality-evaluation-method branch January 22, 2024 22:12
hmaier-fws pushed a commit that referenced this pull request Jan 23, 2024
@jwaspin @dependabot @J-Oliveros
Release v2.8.2 (#275)
6f374f4 
* Bump minimatch and mocha (#263)

Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.1.2 and updates ancestor dependency [mocha](https://github.com/mochajs/mocha). These dependencies need to be updated together.


Updates `minimatch` from 3.0.4 to 3.1.2
- [Release notes](https://github.com/isaacs/minimatch/releases)
- [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
- [Commits](isaacs/minimatch@v3.0.4...v3.1.2)

Updates `mocha` from 5.2.0 to 10.2.0
- [Release notes](https://github.com/mochajs/mocha/releases)
- [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
- [Commits](mochajs/mocha@v5.2.0...v10.2.0)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-type: indirect
- dependency-name: mocha
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add rangeElementDescription.json schema file and updates the attribute.json file to reference it. Example provided.

* fix required

* changes to array of strings and fill out examples

* Remove Old Files (#270)

* Remove Old Files

* Update Version

* Update README.md

* Bugfix - Data Quality (#269)

* Update .gitignore

* Fix Data Quality Evaluation Method

Change from array to single object.

* Update Version

* Update package-lock.json

* Update beta version

* Reordering DQ Report Properties

* Fix Required Data Quality Properties

* Fix Required List

* Fix Report Schema

* Add Additional Enum Options

* Add Required

* Fix Evaluation Method

Changed dateTime to an array.
Added enum options for type.

* Add Required

* Add Required Fields

* Add Descriptions

* add maxItem rule to schema

* Update standaloneQualityReport.json

* Updates

Version numbers.
Requirements.
Add derivedElementReport.
Cleanup rangeElementDescription.

* Update dataQuality.json

Remove max items. It does not appear to do anything.

* Remove beta from version

* Fix Coverage Result

Add qualityResultFile.
Update required fields.

* Fix Coverage Result

resultContent is actually resultContentDescription and is a Coverage Description object not an array.

* Fix Required Fields

---------

Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>
hmaier-fws pushed a commit that referenced this pull request Jan 23, 2024
@jwaspin @dependabot @J-Oliveros
Release v2.8.2 (#275) (#281)
a822180 
* Bump minimatch and mocha (#263)

Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.1.2 and updates ancestor dependency [mocha](https://github.com/mochajs/mocha). These dependencies need to be updated together.


Updates `minimatch` from 3.0.4 to 3.1.2
- [Release notes](https://github.com/isaacs/minimatch/releases)
- [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
- [Commits](isaacs/minimatch@v3.0.4...v3.1.2)

Updates `mocha` from 5.2.0 to 10.2.0
- [Release notes](https://github.com/mochajs/mocha/releases)
- [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
- [Commits](mochajs/mocha@v5.2.0...v10.2.0)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-type: indirect
- dependency-name: mocha
  dependency-type: direct:development
...




* Add rangeElementDescription.json schema file and updates the attribute.json file to reference it. Example provided.

* fix required

* changes to array of strings and fill out examples

* Remove Old Files (#270)

* Remove Old Files

* Update Version

* Update README.md

* Bugfix - Data Quality (#269)

* Update .gitignore

* Fix Data Quality Evaluation Method

Change from array to single object.

* Update Version

* Update package-lock.json

* Update beta version

* Reordering DQ Report Properties

* Fix Required Data Quality Properties

* Fix Required List

* Fix Report Schema

* Add Additional Enum Options

* Add Required

* Fix Evaluation Method

Changed dateTime to an array.
Added enum options for type.

* Add Required

* Add Required Fields

* Add Descriptions

* add maxItem rule to schema

* Update standaloneQualityReport.json

* Updates

Version numbers.
Requirements.
Add derivedElementReport.
Cleanup rangeElementDescription.

* Update dataQuality.json

Remove max items. It does not appear to do anything.

* Remove beta from version

* Fix Coverage Result

Add qualityResultFile.
Update required fields.

* Fix Coverage Result

resultContent is actually resultContentDescription and is a Coverage Description object not an array.

* Fix Required Fields

---------



---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>
hmaier-fws pushed a commit that referenced this pull request Jan 23, 2024
@jwaspin @dependabot @J-Oliveros
Resolve conflicts attempt number 2 (#282)
d01223e 
* Release v2.8.2 (#275)

* Bump minimatch and mocha (#263)

Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.1.2 and updates ancestor dependency [mocha](https://github.com/mochajs/mocha). These dependencies need to be updated together.


Updates `minimatch` from 3.0.4 to 3.1.2
- [Release notes](https://github.com/isaacs/minimatch/releases)
- [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
- [Commits](isaacs/minimatch@v3.0.4...v3.1.2)

Updates `mocha` from 5.2.0 to 10.2.0
- [Release notes](https://github.com/mochajs/mocha/releases)
- [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
- [Commits](mochajs/mocha@v5.2.0...v10.2.0)

---
updated-dependencies:
- dependency-name: minimatch
  dependency-type: indirect
- dependency-name: mocha
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add rangeElementDescription.json schema file and updates the attribute.json file to reference it. Example provided.

* fix required

* changes to array of strings and fill out examples

* Remove Old Files (#270)

* Remove Old Files

* Update Version

* Update README.md

* Bugfix - Data Quality (#269)

* Update .gitignore

* Fix Data Quality Evaluation Method

Change from array to single object.

* Update Version

* Update package-lock.json

* Update beta version

* Reordering DQ Report Properties

* Fix Required Data Quality Properties

* Fix Required List

* Fix Report Schema

* Add Additional Enum Options

* Add Required

* Fix Evaluation Method

Changed dateTime to an array.
Added enum options for type.

* Add Required

* Add Required Fields

* Add Descriptions

* add maxItem rule to schema

* Update standaloneQualityReport.json

* Updates

Version numbers.
Requirements.
Add derivedElementReport.
Cleanup rangeElementDescription.

* Update dataQuality.json

Remove max items. It does not appear to do anything.

* Remove beta from version

* Fix Coverage Result

Add qualityResultFile.
Update required fields.

* Fix Coverage Result

resultContent is actually resultContentDescription and is a Coverage Description object not an array.

* Fix Required Fields

---------

Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

* Re-add package-lock.json

* Update README.md

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>
hmaier-fws added a commit that referenced this pull request Jan 23, 2024
@hmaier-fws
Squashed commit of the following:
185cbac 
commit d01223e
Author: Johnathan <jwaspin@users.noreply.github.com>
Date:   Tue Jan 23 10:39:57 2024 -0700

    Resolve conflicts attempt number 2 (#282)

    * Release v2.8.2 (#275)

    * Bump minimatch and mocha (#263)

    Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.1.2 and updates ancestor dependency [mocha](https://github.com/mochajs/mocha). These dependencies need to be updated together.

    Updates `minimatch` from 3.0.4 to 3.1.2
    - [Release notes](https://github.com/isaacs/minimatch/releases)
    - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
    - [Commits](isaacs/minimatch@v3.0.4...v3.1.2)

    Updates `mocha` from 5.2.0 to 10.2.0
    - [Release notes](https://github.com/mochajs/mocha/releases)
    - [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
    - [Commits](mochajs/mocha@v5.2.0...v10.2.0)

    ---
    updated-dependencies:
    - dependency-name: minimatch
      dependency-type: indirect
    - dependency-name: mocha
      dependency-type: direct:development
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    * Add rangeElementDescription.json schema file and updates the attribute.json file to reference it. Example provided.

    * fix required

    * changes to array of strings and fill out examples

    * Remove Old Files (#270)

    * Remove Old Files

    * Update Version

    * Update README.md

    * Bugfix - Data Quality (#269)

    * Update .gitignore

    * Fix Data Quality Evaluation Method

    Change from array to single object.

    * Update Version

    * Update package-lock.json

    * Update beta version

    * Reordering DQ Report Properties

    * Fix Required Data Quality Properties

    * Fix Required List

    * Fix Report Schema

    * Add Additional Enum Options

    * Add Required

    * Fix Evaluation Method

    Changed dateTime to an array.
    Added enum options for type.

    * Add Required

    * Add Required Fields

    * Add Descriptions

    * add maxItem rule to schema

    * Update standaloneQualityReport.json

    * Updates

    Version numbers.
    Requirements.
    Add derivedElementReport.
    Cleanup rangeElementDescription.

    * Update dataQuality.json

    Remove max items. It does not appear to do anything.

    * Remove beta from version

    * Fix Coverage Result

    Add qualityResultFile.
    Update required fields.

    * Fix Coverage Result

    resultContent is actually resultContentDescription and is a Coverage Description object not an array.

    * Fix Required Fields

    ---------

    Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

    ---------

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

    * Re-add package-lock.json

    * Update README.md

    ---------

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

commit a822180
Author: Johnathan <jwaspin@users.noreply.github.com>
Date:   Tue Jan 23 10:28:41 2024 -0700

    Release v2.8.2 (#275) (#281)

    * Bump minimatch and mocha (#263)

    Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.1.2 and updates ancestor dependency [mocha](https://github.com/mochajs/mocha). These dependencies need to be updated together.

    Updates `minimatch` from 3.0.4 to 3.1.2
    - [Release notes](https://github.com/isaacs/minimatch/releases)
    - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
    - [Commits](isaacs/minimatch@v3.0.4...v3.1.2)

    Updates `mocha` from 5.2.0 to 10.2.0
    - [Release notes](https://github.com/mochajs/mocha/releases)
    - [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
    - [Commits](mochajs/mocha@v5.2.0...v10.2.0)

    ---
    updated-dependencies:
    - dependency-name: minimatch
      dependency-type: indirect
    - dependency-name: mocha
      dependency-type: direct:development
    ...

    * Add rangeElementDescription.json schema file and updates the attribute.json file to reference it. Example provided.

    * fix required

    * changes to array of strings and fill out examples

    * Remove Old Files (#270)

    * Remove Old Files

    * Update Version

    * Update README.md

    * Bugfix - Data Quality (#269)

    * Update .gitignore

    * Fix Data Quality Evaluation Method

    Change from array to single object.

    * Update Version

    * Update package-lock.json

    * Update beta version

    * Reordering DQ Report Properties

    * Fix Required Data Quality Properties

    * Fix Required List

    * Fix Report Schema

    * Add Additional Enum Options

    * Add Required

    * Fix Evaluation Method

    Changed dateTime to an array.
    Added enum options for type.

    * Add Required

    * Add Required Fields

    * Add Descriptions

    * add maxItem rule to schema

    * Update standaloneQualityReport.json

    * Updates

    Version numbers.
    Requirements.
    Add derivedElementReport.
    Cleanup rangeElementDescription.

    * Update dataQuality.json

    Remove max items. It does not appear to do anything.

    * Remove beta from version

    * Fix Coverage Result

    Add qualityResultFile.
    Update required fields.

    * Fix Coverage Result

    resultContent is actually resultContentDescription and is a Coverage Description object not an array.

    * Fix Required Fields

    ---------

    ---------

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

commit 87ee4c2
Author: Johnathan <jwaspin@users.noreply.github.com>
Date:   Tue Jan 23 10:11:24 2024 -0700

    Bugfix/gem version number (#278)

    * Increment Ruby Gem Version

    Incremented to 2.8.2

    * Update package-lock.json

commit 190d591
Merge: c788fee b88a83d
Author: Johnathan Aspinwall <jwaspin@gmail.com>
Date:   Mon Jan 22 15:16:52 2024 -0700

    Merge branch 'dev' into develop

commit b88a83d
Author: Johnathan <jwaspin@users.noreply.github.com>
Date:   Mon Jan 22 12:19:06 2024 -0700

    Bugfix - Data Quality (#269)

    * Update .gitignore

    * Fix Data Quality Evaluation Method

    Change from array to single object.

    * Update Version

    * Update package-lock.json

    * Update beta version

    * Reordering DQ Report Properties

    * Fix Required Data Quality Properties

    * Fix Required List

    * Fix Report Schema

    * Add Additional Enum Options

    * Add Required

    * Fix Evaluation Method

    Changed dateTime to an array.
    Added enum options for type.

    * Add Required

    * Add Required Fields

    * Add Descriptions

    * add maxItem rule to schema

    * Update standaloneQualityReport.json

    * Updates

    Version numbers.
    Requirements.
    Add derivedElementReport.
    Cleanup rangeElementDescription.

    * Update dataQuality.json

    Remove max items. It does not appear to do anything.

    * Remove beta from version

    * Fix Coverage Result

    Add qualityResultFile.
    Update required fields.

    * Fix Coverage Result

    resultContent is actually resultContentDescription and is a Coverage Description object not an array.

    * Fix Required Fields

    ---------

    Co-authored-by: Jonathan Oliveros <joliveros@xentity.com>

commit 0503dc2
Author: Johnathan <jwaspin@users.noreply.github.com>
Date:   Wed Dec 20 13:32:38 2023 -0700

    Remove Old Files (#270)

    * Remove Old Files

    * Update Version

    * Update README.md

commit 7bd1405
Author: Jonathan Oliveros <joliveros@xentity.com>
Date:   Tue Dec 19 15:43:55 2023 -0700

    changes to array of strings and fill out examples

commit 1bd0c93
Author: Jonathan Oliveros <joliveros@xentity.com>
Date:   Tue Dec 19 13:50:58 2023 -0700

    fix required

commit 5749135
Author: Jonathan Oliveros <joliveros@xentity.com>
Date:   Mon Dec 11 16:56:37 2023 -0700

    Add rangeElementDescription.json schema file and updates the attribute.json file to reference it. Example provided.

commit 89e8f2c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Jun 26 18:09:40 2023 -0800

    Bump minimatch and mocha (#263)

    Bumps [minimatch](https://github.com/isaacs/minimatch) to 3.1.2 and updates ancestor dependency [mocha](https://github.com/mochajs/mocha). These dependencies need to be updated together.

    Updates `minimatch` from 3.0.4 to 3.1.2
    - [Release notes](https://github.com/isaacs/minimatch/releases)
    - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md)
    - [Commits](isaacs/minimatch@v3.0.4...v3.1.2)

    Updates `mocha` from 5.2.0 to 10.2.0
    - [Release notes](https://github.com/mochajs/mocha/releases)
    - [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
    - [Commits](mochajs/mocha@v5.2.0...v10.2.0)

    ---
    updated-dependencies:
    - dependency-name: minimatch
      dependency-type: indirect
    - dependency-name: mocha
      dependency-type: direct:development
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hmaier-fws hmaier-fws hmaier-fws approved these changes

@dwalt dwalt Awaiting requested review from dwalt

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@jwaspin @hmaier-fws @J-Oliveros