Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update acceptance test report to include changes in WARNINGS. #1354

Merged
merged 9 commits into from
Mar 17, 2023

Conversation

bdferris-v2
Copy link
Collaborator

Refactor the acceptance test validation report comparison logic to consider changes to WARNING-level notices, not just ERRORs. We could more easily expand this to include INFO-level notices as well, if we so desire. Currently, this change does not cause automatic blocking of a change if the # of warnings changes significantly, but we could change that as well.

As part of this change, I refactored a bunch of logic out of ValidationReport and moved it into a new NoticeSet to be separate some concerns. I also refactored a bunch of references to "errors" to more generic "notices".

Closes #1349.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with gradle test to make sure you didn't break anything
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1420 sources (~0 %) are corrupted.
Commit: ad4009a
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1420 sources (~0 %) are corrupted.
Commit: bd3a8b4
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@bdferris-v2 bdferris-v2 merged commit aa4c5a9 into master Mar 17, 2023
@bdferris-v2 bdferris-v2 deleted the issue/acceptance_test_warnings branch March 17, 2023 18:15
@isabelle-dr
Copy link
Contributor

Do we need to update https://github.com/MobilityData/gtfs-validator/blob/master/docs/ACCEPTANCE_TESTS.md as a result of this PR?

ryon pushed a commit to JarvusInnovations/gtfs-validator that referenced this pull request Apr 1, 2023
…obilityData#1354)

* Track changes to warnings in addition to errors in acceptance tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Acceptance Tests only report changes in ERROR notices, not WARNINGS or INFOs
3 participants