Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: 1537 get list of validators that couldnt run because of a parsing problem #1565

Conversation

qcdyx
Copy link
Contributor

@qcdyx qcdyx commented Sep 7, 2023

Summary:

Closes #1537
My solution goes thru both SingleEntityValidators and SingleFileValidators, gathers validators that have dependencies with parsing errors. From my perspectives, we should decouple this problem into two parts. First, get the list of validators that couldn't run due to parsing errors. Second, a new logic could be added into the gtfs-validator - if we don't have this and that text files, then validator A B C should not run. @emmambd I request all dev team for code review. Let me know your opinion on the idea of decoupling this issue.

Expected behavior:

You will see the following info printing out if there's validators that couldn't run due to parsing errors

| Some validators were skipped due to parsing problems. |

Validators with Parsing Errors: BikesAllowanceValidator,GtfsAttributionRouteIdForeignKeyValidator,GtfsFareLegRuleNetworkIdForeignKeyValidator,GtfsFareRuleRouteIdForeignKeyValidator,GtfsRouteAgencyIdForeignKeyValidator,GtfsTransferFromRouteIdForeignKeyValidator,GtfsTransferToRouteIdForeignKeyValidator,GtfsTripRouteIdForeignKeyValidator,RouteAgencyIdValidator,ShapeToStopMatchingValidator,StopTimeTravelSpeedValidator,TranslationFieldAndReferenceValidator,UrlConsistencyValidator
Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with gradle test to make sure you didn't break anything
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@qcdyx qcdyx linked an issue Sep 7, 2023 that may be closed by this pull request
@qcdyx qcdyx changed the title 1537 get list of validators that couldnt run because of a parsing problem feat: 1537 get list of validators that couldnt run because of a parsing problem Sep 7, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 7, 2023

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Sep 7, 2023

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

@qcdyx qcdyx requested review from jcpitre and cka-y September 7, 2023 20:05
@github-actions
Copy link
Contributor

github-actions bot commented Sep 7, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: 5f72dda
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 8, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: cfc1fc1
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Contributor

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: 47ecf7f
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: ea5f864
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

…tput validators that couldn't run due to parsing problems
@emmambd
Copy link
Contributor

emmambd commented Sep 12, 2023

@qcdyx Can you show an example of how the parsing error output message will look now / give me instructions on how to generate it? This will help with QA.

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: 2c5468f
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Contributor

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

1 similar comment
@github-actions
Copy link
Contributor

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: 3e810ac
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Contributor

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: ec03150
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@jcpitre
Copy link
Contributor

jcpitre commented Sep 14, 2023

@emmambd I created a dataset where the agency.txt file did not have an agency name. Here is what is printed:

---------------------------------------------------------------------------------------- 
| Some validators were skipped because the GTFS files they rely on could not be parsed | 
---------------------------------------------------------------------------------------- 
Multi-file validators:
   FareAttributeAgencyIdValidator
   GtfsAttributionAgencyIdForeignKeyValidator
   GtfsFareAttributeAgencyIdForeignKeyValidator
   GtfsRouteAgencyIdForeignKeyValidator
   MatchingFeedAndAgencyLangValidator
   RouteAgencyIdValidator
   TranslationFieldAndReferenceValidator
   UrlConsistencyValidator

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: 35331e1
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Contributor

✅ Rule acceptance tests passed.
New Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1470 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1470 sources (~0 %) are corrupted.
Commit: 175375e
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@qcdyx qcdyx merged commit 57eed2f into master Sep 15, 2023
334 checks passed
@qcdyx qcdyx deleted the 1537-get-list-of-validators-that-couldnt-run-because-of-a-parsing-problem branch September 15, 2023 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Get list of validators that couldn't run because of a parsing problem
3 participants