3.0.0 Release tracking

[isabelle-dr]

This is an issue for the 3.0.0 release tracking. It was initially planned for the end beginning October 2021.
For the exhaustive list of everything included in this release, please refer to the 3.0.0 Milestone tag.
Update November 9, 2021: the v3.0.0 is out in pre-release! 🎉
Update November 29, 2021: for the first time, we ran analytics on all the datasets available on the mobility database! Results here.
Update January 10, 2022: the last PR is merged, which wraps up this release, better late than never!

Goals for the release 🎯

1. Solve discrepancies with the official GTFS specification. This has been done after the update to RFC 2119 has been done. The V2.0.0 to V3.0.0 rule mapping is available here.
2. Implement automated tests to see if a Pull Request would result in datasets from the mobility database getting additional errors.
3. Make sure MobilityData's work has related public communication to facilitate community contribution (issues opened, public sprint board and bug tracking board, roadmap, ways for users to voice what they'd like to see next and for contributors to see how to get involved).

Work ⚙️

1. Solve discrepancies with the spec

• Make LeadingOrTrailingWhitespacesNotice a WARNING instead of an ERROR #920
  • chore: downgrade LeadingOrTrailingWhitespacesNotice to WARNING #929
• Add stops.stop_url != (agency.agency_url or routes.route_url) #909
  • chore: new rule - additional requirements urls (stops.txt, routes.txt, agency.txt) #932)
• Add routes.route_url != agency.agency_url #907
  • chore: new rule - additional requirements urls (stops.txt, routes.txt, agency.txt) #932

stops.txt

• feat: Validate presence of location_type and the type of referenced location #500
• Add validation on stops.stop_desc != stops.stop_name #910
  • chore: check stop name and desc #937
• Implement Zone ID presence verification (GTFS Rule) #237
  • chore: check stops.zone_id presence #936
• chore: upgrade StopTimeTimepointWithoutTimesNotice to ERROR #1068

stop_times.txt

• Check allowed location_types for stops referenced by stop_times.stop_id  #763
  • chore: new rule - additional check on stops.location_type #938

pathways.txt

• new rule: attach pathways to boarding areas' of platforms #980
  • feat: Validate that pathway endpoints have correct types #961
  • feat: Validate that a pathway is not a loop #999
• new rule: no locked platforms #975
  • feat: Validate that locations are reachable if a station has pathways #997
• new rule: No dangling locations in pathways.txt #974
  • feat: Validate that a generic node is not dangling #998

translations.txt

• Check conditional requirement on translations.txt fields #820
  • feat: Validate field presence and key integrity for translations.txt #968

levels.txt

• Add validation on levels.txt presence #921
  • feat: additional checks on pathways.txt and stops.txt #967
  • feat: verify conditional requirement of levels.txt presence #966
  • refactor: rename MissingLevelNotice -> MissingLevelFileNotice #969

2. Automated test to see if a PR results in additional errors

• Automated tests to see if a PR will trigger new errors in datasets #1045
  • chore: get urls of the database as a list of 256 objets max #1037
  • ci: implement queuing mechanism for acceptance tests #1038
  • feat: Adds Python script and requirements to harvest the latest versions (acceptance tests) #926
  • feat: acceptance tests #848

3. Public communication

• Public sprint board
• Publig bug tracking board, populated automatically
• Public Release plan / release tracking issue (this one!)
• High level Roadmap in a ProductBoard portal (link coming shortly)

Other work included

Better rules

• feat: Implement ShapeToStopMatchingValidator #956
• feat: Check that vehicles do not travel too fast #944
• feat: Serialize S2LatLng fields in Notice subclasses #958
• chore: Protect against infinite loop in getStopOrParentLatLng #957

Better user experience

• Ability to dump validator notice schemas to json or yaml #899
  • feat: export notice schema as json file #925
  • feat: Add NoticeSchemaGenerator #951
  • feat: Return JsonObject from NoticeContainer methods #952
• Make -f an optional command-line parameter #539
  • feat!: deprecate -f  #851
• Add command-line parameter to output "pretty" JSON #838
  • feat: introduce -p CLI parameter to output beautified reports #901
• add help as CLI parameter #857
  • feat: print help menu #888

Bugs

• NoticeContainer should impose limit on the number of added notices per notice type during validation #959
• False positives for StopTimeTimepointWithoutTimesNotice #954
• NullPointerException at org.mobilitydata.gtfsvalidator.validator.ParentLocationTypeValidator.validate(ParentLocationTypeValidator.java:64) #1010
• fix: Skip phone number validation when country code is unknown #1062

Other good stuff

• chore: Upgrade to Java 11 #1014
• ci: check code formatting on pull requests #953
• chore: upgrade to gradle wrapper to 7.2 #965
  ............... and much more

Contributors 🙏

Massive shoutout to:
🎉 @lionel-nj
🎉 @maximearmstrong
🎉 @barbeau
🎉 @aababilov
🎉 @asvechnikov2

[isabelle-dr]

Update November 9, 2021: the v3.0.0-beta is out! 🎉

[markstos]

I tried the 3.0 beta. It worked pretty well, but declared a phone number in the phone number 111-222-333 as invalid, although according to the spec, it seems like should be valid:

https://developers.google.com/transit/gtfs/reference#agencytxt

I believe this issue may already be fixed now by: #1062 but is not included in the beta release.

[isabelle-dr]

Hi @markstos , the issue has been fixed, you can use this jar file to run the version of the validator where this issue is fixed!

[isabelle-dr]

The last PR is now merged, closing this issue