RFC for gtfs extension #111

schlingling · 2023-01-22T18:23:04Z

See rfc/0002-gtfs-extension.md for detailled explanation. Really looking foward to your feedback!

github-actions · 2023-01-22T18:23:19Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

schlingling · 2023-01-22T18:26:42Z

CLA Assistant Lite bot: Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

You can retrigger this bot by commenting recheck in this Pull Request

CLA already submitted by physical letter (should arrive in Erlangen soon)

rhazn · 2023-01-22T18:30:55Z

The bot will not give you piece until you comment the "I have read the CLA Document and I hereby sign the CLA" and trigger a recheck, sorry. Would be great if you can do that in addition to the signed CLA. Thanks, will look at this early next week!

schlingling · 2023-01-22T20:39:55Z

I have read the CLA Document and I hereby sign the CLA

schlingling · 2023-01-22T20:40:42Z

Recheck

rhazn

Very well written RFC I think, thank you. I left some notes, @felix-oq would you be so nice and do a review as well? If anything is unclear, feel free to ping me on slack. FYI @georg-schwarz as well, happy if you have input but not a requirement.

rfc/0002-gtfs-extension.md

rhazn · 2023-01-25T11:58:57Z

rfc/0002-gtfs-extension.md

+Disclaimer: This RFC is part of my master-thesis "Archiving open transport data using the JValue tooling ecosystem" supervised by @rhazn. I'll open an PR for that, in order to discuss the neccessary changes to Jayvee due to my implemenation.
+
+# Summary
+This RFC enables a pipeline extracting, validating and loading GTFS-data by providing an GTFS-endpoint under consideration of the [GTFS-speficiation](https://developers.google.com/transit/gtfs/reference). For that reason some changes and extensions of Jayvee have to be made. The overall goal of this RFC is processing GTFS-data with a minimun of changes/extensions in Jayvee. 


"GTFS-specification", "minimum"

fixed these typos

rhazn · 2023-01-25T12:04:32Z

rfc/0002-gtfs-extension.md

+
+Each of the following subchapters explains the idea behind.
+
+## 1) HttpExtractor


I like it, should be part of the std extension.

TODO: How to deal with that? When i start implementation, should i create them already in std-folder or first in extensions->mobility and we move this later?

I think you can implement these in the std folder.

rhazn · 2023-01-25T12:04:37Z

rfc/0002-gtfs-extension.md

+}
+```
+
+## 2) ZipInterpreter


I like it, should be part of the std extension.

TODO: How to deal with that? When i start implementation, should i create them already in std-folder or first in extensions->mobility and we move this later?

rhazn · 2023-01-25T12:05:32Z

rfc/0002-gtfs-extension.md

+}
+```
+
+## 3) GtfsInterpreter


I like it, collections of sheets should be part of the std extension.

This block, however, should be part of the new extension.

Okay, i will implement the GtfsInterpreter in the mobility-extension
TODO: Lets discuss in our call, where to implement collections exactly (i think, in the io-types.ts file).

rhazn · 2023-01-25T12:15:55Z

rfc/0002-gtfs-extension.md

+```
+
+### 4.1) Layout
+As GTFS defines a whole datamodel consisting of multiple dimensions, we can provide (in some future) the user with a predefined set of layouts. Some columns in GTFS-csv-files are optional and conditional optional, what can be reflected in the grammar using `?` and `!` as an flag. From a generic perspective, these flags can be considered in every Validator, depending on the implementation, since optional columns could be in all kinds of csv-files not just in gtfs-csv-files. A vision is, that the GTFS-pipeline later on processes a list of GTFS-Endpoints. Because every endpoint has at least the required-columns, we need to have the optional-mechanism in our layout.


I think we should represent the optional columns with their datatype, e.g. saying their datatype is text or undefined. For that, we'd need an undefined datatype in Jayvee and a way to combine these types.

For this RFC, I would only include required columns and ignore everything else.

I will add a chapter, describing this behavior

rhazn · 2023-01-25T12:17:44Z

rfc/0002-gtfs-extension.md

+
+In a GTFS-Validator, some conditional (aka logical) checks could possibly be applied during the validation (not just a static header/datatype validaton)
+
+In the specifiation, the order of the columns is not defined, so we need to access the columns by their names, not their index as every GTFS-endpoint could possibly have a different order!! 


Very important and interesting requirement, I think we should let this influence #114 directly. I'll mention it there. I could see selecting cell ranges by names, aka column A or column "agency_id" (note the quotation marks to show it is a string).

rhazn · 2023-01-25T12:21:48Z

rfc/0002-gtfs-extension.md

+### 4.2) Layouts
+As we want to process a zip-file containing multiple csv-files, we also need a mechanism to map csv-files to layouts. In my [.jv-file](0002-gtfs.jv), every layout gets defined as usual (header, column, datatypes --> please have look at chapter 4.1). Multiple layouts get then wrapped in a new grammar-feature, which maps csv-filenames to layouts. This layouts-artefact gets referenced by `SQLiteTablesLoader` and `LayoutsValidator`, too.
+```
+layouts gtfsLayouts {


I am not sure if we need a new grammar feature for this. We definitely should talk about how to express collections with and without string keys in our language (and if we want to).

Originally, we had an idea to consider an attribute that was added multiple times as a collection. For collections with a string index, we'd use <key> with <value> syntax for each element, for arrays just the value. In this case, it could be:

block GtfsValidator oftype LayoutsValidator { validationLayouts: "agency" with agencyLayout; validationLayouts: "stops" with stopsLayout; }

An alternative could be to allow key, value tuples, consider:

block GtfsValidator oftype LayoutsValidator { validationLayouts: ("agency", agencyLayout); validationLayouts: ("stops", stopsLayout); }

Lets discuss this in our call

rhazn · 2023-01-25T12:24:06Z

rfc/0002-gtfs-extension.md

+
+```
+block GtfsLoader oftype SQLiteTablesLoader {
+    tables: gtfsLayouts;


At this point, I would honestly just assume the keys of the collection can be used as table names for now, write that down and remove the tables attribute here:

block GtfsLoader oftype SQLiteTablesLoader { file: "./gtfs.db"; }

rhazn · 2023-01-25T12:28:26Z

rfc/0002-gtfs.jv

+
+	layout agencyLayout {
+		header row 1: text;
+		column agency_id!: text; //Conditional optional columns marked with ! (but could be any other literal as well... i took that one because ? stands for optional)


How can this be conditionally optional if routes have an agency_id that is not optional? Is this optional if no routes file (or other file that has agency_id as required) exists?

In any case, as discussed before, I would remove the !/? logic for now and just consider the required columns + consider this cond. required column as required.

How can this be conditionally optional if routes have an agency_id that is not optional? Is this optional if no routes file (or other file that has agency_id as required) exists?

A dataset may contain data from multiple agencies. This field is required when the dataset contains data for multiple transit agencies, otherwise it is optional.

I remove the ! ? stuff in the new commit.

schlingling

I apply all changes mentioned directly and will add a new commit. Should i resolve all comments, where no more action is needed @rhazn?

schlingling · 2023-01-25T16:24:33Z

rfc/0002-gtfs-extension.md

+}
+```
+
+## 2) ZipInterpreter


TODO: How to deal with that? When i start implementation, should i create them already in std-folder or first in extensions->mobility and we move this later?

schlingling · 2023-01-25T16:29:17Z

rfc/0002-gtfs-extension.md

+}
+```
+
+## 3) GtfsInterpreter


Okay, i will implement the GtfsInterpreter in the mobility-extension
TODO: Lets discuss in our call, where to implement collections exactly (i think, in the io-types.ts file).

schlingling · 2023-01-25T16:35:05Z

rfc/0002-gtfs-extension.md

+}
+```
+
+## 4) LayoutsValidator


I will add a chapter for describing the new io-datatypes needed

schlingling · 2023-01-25T16:35:59Z

rfc/0002-gtfs-extension.md

+```
+
+### 4.1) Layout
+As GTFS defines a whole datamodel consisting of multiple dimensions, we can provide (in some future) the user with a predefined set of layouts. Some columns in GTFS-csv-files are optional and conditional optional, what can be reflected in the grammar using `?` and `!` as an flag. From a generic perspective, these flags can be considered in every Validator, depending on the implementation, since optional columns could be in all kinds of csv-files not just in gtfs-csv-files. A vision is, that the GTFS-pipeline later on processes a list of GTFS-Endpoints. Because every endpoint has at least the required-columns, we need to have the optional-mechanism in our layout.


I will add a chapter, describing this behavior

schlingling · 2023-01-25T16:42:06Z

rfc/0002-gtfs-extension.md

+### 4.2) Layouts
+As we want to process a zip-file containing multiple csv-files, we also need a mechanism to map csv-files to layouts. In my [.jv-file](0002-gtfs.jv), every layout gets defined as usual (header, column, datatypes --> please have look at chapter 4.1). Multiple layouts get then wrapped in a new grammar-feature, which maps csv-filenames to layouts. This layouts-artefact gets referenced by `SQLiteTablesLoader` and `LayoutsValidator`, too.
+```
+layouts gtfsLayouts {


Lets discuss this in our call

schlingling · 2023-01-25T16:44:05Z

rfc/0002-gtfs-extension.md

+
+```
+block GtfsLoader oftype SQLiteTablesLoader {
+    tables: gtfsLayouts;


schlingling · 2023-01-25T16:47:32Z

rfc/0002-gtfs.jv

+
+	layout agencyLayout {
+		header row 1: text;
+		column agency_id!: text; //Conditional optional columns marked with ! (but could be any other literal as well... i took that one because ? stands for optional)


How can this be conditionally optional if routes have an agency_id that is not optional? Is this optional if no routes file (or other file that has agency_id as required) exists?

A dataset may contain data from multiple agencies. This field is required when the dataset contains data for multiple transit agencies, otherwise it is optional.

I remove the ! ? stuff in the new commit.

schlingling · 2023-02-26T10:03:33Z

This PR is a basis for #123

schlingling added 2 commits January 22, 2023 16:11

added example for gtfs specific jayvee-file

022c004

added rfc-0002 description and resources

01d306c

schlingling added the enhancement New feature or request label Jan 22, 2023

schlingling requested a review from rhazn January 22, 2023 18:23

schlingling self-assigned this Jan 22, 2023

fixed typos

99e479b

rhazn reviewed Jan 25, 2023

View reviewed changes

schlingling requested a review from felix-oq January 25, 2023 16:01

schlingling commented Jan 25, 2023

View reviewed changes

schlingling changed the title ~~RFC for GTFS extension~~ RFC for mobility extension Jan 25, 2023

added explanation for io-datatypes, optional-mechanism etc.

72ed016

schlingling closed this Jan 25, 2023

schlingling deleted the rfc-gtfs-extension branch January 25, 2023 17:49

github-actions bot locked and limited conversation to collaborators Jan 25, 2023

schlingling changed the title ~~RFC for mobility extension~~ RFC for gtfs extension Jan 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC for gtfs extension #111

RFC for gtfs extension #111

schlingling commented Jan 22, 2023

github-actions bot commented Jan 22, 2023 •

edited

schlingling commented Jan 22, 2023

rhazn commented Jan 22, 2023

schlingling commented Jan 22, 2023 •

edited

schlingling commented Jan 22, 2023

rhazn left a comment

rhazn Jan 25, 2023

schlingling Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023 •

edited

rhazn Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023

rhazn Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023

rhazn Jan 25, 2023

schlingling Jan 25, 2023

schlingling left a comment •

edited

schlingling Jan 25, 2023

schlingling Jan 25, 2023

schlingling Jan 25, 2023

schlingling Jan 25, 2023

schlingling Jan 25, 2023

schlingling Jan 25, 2023

schlingling Jan 25, 2023

schlingling commented Feb 26, 2023


		Each of the following subchapters explains the idea behind.

		## 1) HttpExtractor


		In a GTFS-Validator, some conditional (aka logical) checks could possibly be applied during the validation (not just a static header/datatype validaton)

		In the specifiation, the order of the columns is not defined, so we need to access the columns by their names, not their index as every GTFS-endpoint could possibly have a different order!!

RFC for gtfs extension #111

RFC for gtfs extension #111

Conversation

schlingling commented Jan 22, 2023

github-actions bot commented Jan 22, 2023 • edited

schlingling commented Jan 22, 2023

rhazn commented Jan 22, 2023

schlingling commented Jan 22, 2023 • edited

schlingling commented Jan 22, 2023

rhazn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schlingling Jan 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schlingling left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schlingling commented Feb 26, 2023

github-actions bot commented Jan 22, 2023 •

edited

schlingling commented Jan 22, 2023 •

edited

schlingling Jan 25, 2023 •

edited

schlingling left a comment •

edited