feat: Initial support for GTFS Fares v2 - fare media

[bdferris-v2]

As described in google/transit#355. This is a WIP that is intended to track the proposed modifications the spec. Closes #1301.

Please make sure these boxes are checked before submitting your pull request - thanks!

• Run the unit tests with gradle test to make sure you didn't break anything
• Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
• Linked all relevant issues

[isabelle-dr]

@bdferris-v2 Thanks for working on this 🙏! Can we make this fork available for stakeholders producing this data?
Here are the name changes that we've made after you created this PR:

• fare_payment_types.txt -> fare_payment_options.txt
• fare_payment_type_group_id -> fare_payment_option_group_id
• fare_payment_type_name -> fare_payment_option_name
• fare_payment_type -> fare_payment_option_type

[bdferris-v2]

@isabelle-dr done

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 0 out of 1419 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1419 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1419 sources (~0 %) are corrupted.
Commit: 2965826
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 1 out of 1419 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1419 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1419 sources (~0 %) are corrupted.
Commit: 089ad3e
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[isabelle-dr]

@bdferris-v2 this is now unblocked :)
I believe we only need to update medium -> media in this PR.

I suggest we keep the paper_ticket as well, I expect it to be adopted soon and I see more pros (avoid false positives, avoid confusion) than cons (not yet adopted) in keeping it in this validator.
Let me know if you think otherwise.

[bdferris-v2]

@isabelle-dr I had already updated the bulk of references - just had to fix a few lingering mentions in the docs. This is ready for review.

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1420 sources (~0 %) are corrupted.
Commit: e2d33bf
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[isabelle-dr]

@bdferris-v2 there seems to be a problem with the structure of the duplicate_fare_media notice. See the image of the report.

Here is the JSON if it helps:

{"code":"duplicate_fare_media",
  "severity":"WARNING",
  "totalNotices":1,
  "sampleNotices":[{
    "fareMedia1":{
        "csvRowNumber":6.0,"fareMediaId":"cash"},
      "fareMedia2":{
        "csvRowNumber":7.0,"fareMediaId":"cash2"}}]}

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1420 sources (~0 %) are corrupted.
Commit: bce6bd8
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[bdferris-v2]

So back to the high-level question from @isabelle-dr. I'll admit I took a slightly different approach to Notice schema modeling here and I wanted to explain why.

We have lots of notices where we are documenting the differences between two entities in a file. The existing convention for notices has been to introduce fields like:

• csvRowNumber1
• filename1
• fieldName1

and corresponding:

• csvRowNumber2
• filename2
• fieldName2

While that works, it leads to duplication of field descriptions/documentation and logic for setting field values.

Here, I took a different approach, where I bundled the fields for an entity together (csvRowNumber and fareMediaId) and then duplicated those across two parent fields fareMedia1 and fareMedia2.

The resulting JSON schema looks like:

{
  "fareMedia1": { "csvRowNumber": 1, "fareMediaId": "id1" },
  "fareMedia2": { "csvRowNumber": 2, "fareMediaId": "id2" }
}

I'll admit that renders a little weirdly in the existing report UI, but I have a follow-up PR that would make it render like this:

Note the nested rows of field name headers at the top.

Thoughts?

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1423 sources (~0 %) are corrupted.
Commit: c334e58
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[isabelle-dr]

Okay, I understand. This seems more straightforward indeed.

I am wondering if the users that programmatically consume the JSON report will have issues with this.
@lauriemerrell and @derhuerst do you foresee any issues with having this additional nesting in the JSON report?

[lauriemerrell]

As noted on Slack, @atvaccaro will review for Cal-ITP impacts; thanks for the call-out @isabelle-dr, this does affect us!

[atvaccaro]

As noted on Slack, @atvaccaro will review for Cal-ITP impacts; thanks for the call-out @isabelle-dr, this does affect us!

Cross-posting from Slack: would it be possible to stick with either totally unnested or totally nested notices? Right we just parse each record as flat i.e. no nesting (https://github.com/cal-itp/data-infra/blob/main/airflow/dags/create_external_tables/gtfs_schedule_v2/validation_notices.yml#L41-L299) and while I wouldn’t be opposed to the nesting in general, my understanding is that the PR would only change to nesting for this specific fares work. (please correct me if I’m wrong)

We can handle nested fields for sure, just wondering about overall consistency.

[bdferris-v2]

It's true that this change is only for the single fare-related notice. There are a handful of existing notices that I believe could benefit from a nested field approach:

duplicate_key
foreign_key_violation
block_trips_with_overlapping_stop_times
duplicate_route_name
overlapping_frequency
wrong_parent_location_type
decreasing_shape_distance
equal_shape_distance_diff_coordinates
equal_shape_distance_same_coordinates
stops_match_shape_out_of_order
decreasing_or_equal_stop_time_distance
fast_travel_between_consecutive_stops
fast_travel_between_far_stops

but I acknowledge it's a more significant decision to migrate all of them and probably one we couldn't make in time for the next release. And if we were going to make a breaking change to the notice schema, I suppose I'd want to use it as an opportunity to fix some other inconsistencies as well (csvRowNumber1 vs csvRowNumberA vs oldCsvRowNumber, etc).

So I guess that's a long way of saying that I will revert the nest field notices in this PR for now.

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1423 sources (~0 %) are corrupted.
Commit: d58bb44
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[derhuerst]

While I don't like that we're adding a different format/schema just for our use case here, I agree that breaking backwards compatibility (by changing all other notices' formats to work this way too) should be done in a more coordinated fashion. Si I'm okay with the format. 👍

[derhuerst]

I am wondering if the users that programmatically consume the JSON report will have issues with this. […] @derhuerst do you […]?

I should note that I do not parse GTFS Validator's output in production yet. I have built a quick prototype script once, and I'm planning to use it in production, but not yet. 😊

[isabelle-dr]

Thank you for the feedback.
@bdferris-v2, I've updated the notice schema in the doc to represent your latest changes.
Merging this PR, thank you!

[github-actions]

✅ Rule acceptance tests passed.
New Errors: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 1 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1423 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1423 sources (~0 %) are corrupted.
Commit: 81dfd0f
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

[atvaccaro]

It's true that this change is only for the single fare-related notice. There are a handful of existing notices that I believe could benefit from a nested field approach:
...
And if we were going to make a breaking change to the notice schema, I suppose I'd want to use it as an opportunity to fix some other inconsistencies as well (csvRowNumber1 vs csvRowNumberA vs oldCsvRowNumber, etc).

Sounds good to me! We'd definitely be on board with normalization and consistency fixes, but agree that doing it in a single breaking change would be ideal.