[UAC-20] Jayvee processes successfully mobility.jv

[schlingling]

This PR is the final one of [FEATURE] Mobility Extension (RFC0002) #123 and covers:

• Fixing typos in mobility.jv
• Changing pipeline to process sample-feed provided by google
• Fixing archive-processing
• Adding arrow-syntax to mobility.jv for better readability

Part of RFC0002 mobility-extension.

Implements following UACs of #123:

• [UAC-20] Jayvee processes successfully mobility.jv

Jayvee now successfully processes a gtfs-feed, which can be seen in following logfile as well as the schema of the sqlite-sink after execution:

Jayvee Log

Pipeline GtfsPipeline:
        Runtime Parameters (0):
        Blocks (46 blocks with 11 pipes):
         -> MyHttpExtractor (HttpExtractor)
                 -> ZipArchiveInterpreter (ArchiveInterpreter)
                         -> AgencyFilePicker (FilePicker)
                                 -> AgencyCSVInterpreter (CSVInterpreter)
                                         -> AgencyTableInterpreter (TableInterpreter)
                                                 -> AgencyLoader (SQLiteLoader)
                         -> CalendarDatesFilePicker (FilePicker)
                                 -> CalendarDatesCSVInterpreter (CSVInterpreter)
                                         -> CalendarDatesTableInterpreter (TableInterpreter)
                                                 -> CalendarDatesLoader (SQLiteLoader)
                         -> CalendarFilePicker (FilePicker)
                                 -> CalendarCSVInterpreter (CSVInterpreter)
                                         -> CalendarTableInterpreter (TableInterpreter)
                                                 -> CalendarLoader (SQLiteLoader)
                         -> FareAttributesFilePicker (FilePicker)
                                 -> FareAttributesCSVInterpreter (CSVInterpreter)
                                         -> FareAttributesTableInterpreter (TableInterpreter)
                                                 -> FareAttributesLoader (SQLiteLoader)
                         -> FareRulesFilePicker (FilePicker)
                                 -> FareRulesCSVInterpreter (CSVInterpreter)
                                         -> FareRulesTableInterpreter (TableInterpreter)
                                                 -> FareRulesLoader (SQLiteLoader)
                         -> FrequenciesFilePicker (FilePicker)
                                 -> FrequenciesCSVInterpreter (CSVInterpreter)
                                         -> FrequenciesTableInterpreter (TableInterpreter)
                                                 -> FrequenciesLoader (SQLiteLoader)
                         -> RoutesFilePicker (FilePicker)
                                 -> RoutesCSVInterpreter (CSVInterpreter)
                                         -> RoutesTableInterpreter (TableInterpreter)
                                                 -> RoutesLoader (SQLiteLoader)
                         -> ShapesFilePicker (FilePicker)
                                 -> ShapesCSVInterpreter (CSVInterpreter)
                                         -> ShapesTableInterpreter (TableInterpreter)
                                                 -> ShapesLoader (SQLiteLoader)
                         -> StopTimesFilePicker (FilePicker)
                                 -> StopTimesCSVInterpreter (CSVInterpreter)
                                         -> StopTimesTableInterpreter (TableInterpreter)
                                                 -> StopTimesLoader (SQLiteLoader)
                         -> StopsFilePicker (FilePicker)
                                 -> StopsCSVInterpreter (CSVInterpreter)
                                         -> StopsTableInterpreter (TableInterpreter)
                                                 -> StopsLoader (SQLiteLoader)
                         -> TripsFilePicker (FilePicker)
                                 -> TripsCSVInterpreter (CSVInterpreter)
                                         -> TripsTableInterpreter (TableInterpreter)
                                                 -> TripsLoader (SQLiteLoader)
[MyHttpExtractor] Fetching raw data from https://developers.google.com/static/transit/gtfs/examples/sample-feed.zip
[MyHttpExtractor] Successfully fetched raw data
[ZipArchiveInterpreter] Loading zip file from binary content
[TripsCSVInterpreter] Parsing raw data as CSV using delimiter ","
[TripsCSVInterpreter] Parsing raw data as CSV-sheet successfull
[TripsTableInterpreter] Matching header with provided column names
[TripsTableInterpreter] Validating 11 row(s) according to the column types
[TripsTableInterpreter] Validation completed, the resulting table has 11 row(s) and 7 column(s)
[TripsLoader] Opening database file ./sample-feed.db
[TripsLoader] Dropping previous table "trips" if it exists
[TripsLoader] Creating table "trips"
[TripsLoader] Inserting 11 row(s) into table "trips"
[TripsLoader] The data was successfully loaded into the database
[StopsCSVInterpreter] Parsing raw data as CSV using delimiter ","
[StopsCSVInterpreter] Parsing raw data as CSV-sheet successfull
[StopsTableInterpreter] Matching header with provided column names
[StopsTableInterpreter] Validating 9 row(s) according to the column types
[StopsTableInterpreter] Validation completed, the resulting table has 9 row(s) and 7 column(s)
[StopsLoader] Opening database file ./sample-feed.db
[StopsLoader] Dropping previous table "stops" if it exists
[StopsLoader] Creating table "stops"
[StopsLoader] Inserting 9 row(s) into table "stops"
[StopsLoader] The data was successfully loaded into the database
[StopTimesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[StopTimesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[StopTimesTableInterpreter] Matching header with provided column names
[StopTimesTableInterpreter] Validating 28 row(s) according to the column types
[StopTimesTableInterpreter] The value at cell F17 does not match the type text
[StopTimesTableInterpreter] The value at cell G17 does not match the type text
[StopTimesTableInterpreter] The value at cell H17 does not match the type text
[StopTimesTableInterpreter] The value at cell I17 does not match the type text
[StopTimesTableInterpreter] Omitting row 17
[StopTimesTableInterpreter] The value at cell F18 does not match the type text
[StopTimesTableInterpreter] The value at cell G18 does not match the type text
[StopTimesTableInterpreter] The value at cell H18 does not match the type text
[StopTimesTableInterpreter] The value at cell I18 does not match the type text
[StopTimesTableInterpreter] Omitting row 18
[StopTimesTableInterpreter] The value at cell F19 does not match the type text
[StopTimesTableInterpreter] The value at cell G19 does not match the type text
[StopTimesTableInterpreter] The value at cell H19 does not match the type text
[StopTimesTableInterpreter] The value at cell I19 does not match the type text
[StopTimesTableInterpreter] Omitting row 19
[StopTimesTableInterpreter] The value at cell F20 does not match the type text
[StopTimesTableInterpreter] The value at cell G20 does not match the type text
[StopTimesTableInterpreter] The value at cell H20 does not match the type text
[StopTimesTableInterpreter] The value at cell I20 does not match the type text
[StopTimesTableInterpreter] Omitting row 20
[StopTimesTableInterpreter] The value at cell F21 does not match the type text
[StopTimesTableInterpreter] The value at cell G21 does not match the type text
[StopTimesTableInterpreter] The value at cell H21 does not match the type text
[StopTimesTableInterpreter] The value at cell I21 does not match the type text
[StopTimesTableInterpreter] Omitting row 21
[StopTimesTableInterpreter] The value at cell F22 does not match the type text
[StopTimesTableInterpreter] The value at cell G22 does not match the type text
[StopTimesTableInterpreter] The value at cell H22 does not match the type text
[StopTimesTableInterpreter] The value at cell I22 does not match the type text
[StopTimesTableInterpreter] Omitting row 22
[StopTimesTableInterpreter] The value at cell F23 does not match the type text
[StopTimesTableInterpreter] The value at cell G23 does not match the type text
[StopTimesTableInterpreter] The value at cell H23 does not match the type text
[StopTimesTableInterpreter] The value at cell I23 does not match the type text
[StopTimesTableInterpreter] Omitting row 23
[StopTimesTableInterpreter] The value at cell F24 does not match the type text
[StopTimesTableInterpreter] The value at cell G24 does not match the type text
[StopTimesTableInterpreter] The value at cell H24 does not match the type text
[StopTimesTableInterpreter] The value at cell I24 does not match the type text
[StopTimesTableInterpreter] Omitting row 24
[StopTimesTableInterpreter] The value at cell F25 does not match the type text
[StopTimesTableInterpreter] The value at cell G25 does not match the type text
[StopTimesTableInterpreter] The value at cell H25 does not match the type text
[StopTimesTableInterpreter] The value at cell I25 does not match the type text
[StopTimesTableInterpreter] Omitting row 25
[StopTimesTableInterpreter] The value at cell F26 does not match the type text
[StopTimesTableInterpreter] The value at cell G26 does not match the type text
[StopTimesTableInterpreter] The value at cell H26 does not match the type text
[StopTimesTableInterpreter] The value at cell I26 does not match the type text
[StopTimesTableInterpreter] Omitting row 26
[StopTimesTableInterpreter] The value at cell F27 does not match the type text
[StopTimesTableInterpreter] The value at cell G27 does not match the type text
[StopTimesTableInterpreter] The value at cell H27 does not match the type text
[StopTimesTableInterpreter] The value at cell I27 does not match the type text
[StopTimesTableInterpreter] Omitting row 27
[StopTimesTableInterpreter] The value at cell F28 does not match the type text
[StopTimesTableInterpreter] The value at cell G28 does not match the type text
[StopTimesTableInterpreter] The value at cell H28 does not match the type text
[StopTimesTableInterpreter] The value at cell I28 does not match the type text
[StopTimesTableInterpreter] Omitting row 28
[StopTimesTableInterpreter] The value at cell F29 does not match the type text
[StopTimesTableInterpreter] The value at cell G29 does not match the type text
[StopTimesTableInterpreter] The value at cell H29 does not match the type text
[StopTimesTableInterpreter] The value at cell I29 does not match the type text
[StopTimesTableInterpreter] Omitting row 29
[StopTimesTableInterpreter] Validation completed, the resulting table has 15 row(s) and 9 column(s)
[StopTimesLoader] Opening database file ./sample-feed.db
[StopTimesLoader] Dropping previous table "stop_times" if it exists
[StopTimesLoader] Creating table "stop_times"
[StopTimesLoader] Inserting 15 row(s) into table "stop_times"
[StopTimesLoader] The data was successfully loaded into the database
[ShapesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[ShapesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[ShapesTableInterpreter] Matching header with provided column names
[ShapesTableInterpreter] Validating 0 row(s) according to the column types
[ShapesTableInterpreter] Validation completed, the resulting table has 0 row(s) and 5 column(s)
[ShapesLoader] Opening database file ./sample-feed.db
[ShapesLoader] Dropping previous table "shapes" if it exists
[ShapesLoader] Creating table "shapes"
[ShapesLoader] Inserting 0 row(s) into table "shapes"
[ShapesLoader] The data was successfully loaded into the database
[RoutesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[RoutesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[RoutesTableInterpreter] Matching header with provided column names
[RoutesTableInterpreter] Validating 5 row(s) according to the column types
[RoutesTableInterpreter] Validation completed, the resulting table has 5 row(s) and 9 column(s)
[RoutesLoader] Opening database file ./sample-feed.db
[RoutesLoader] Dropping previous table "routes" if it exists
[RoutesLoader] Creating table "routes"
[RoutesLoader] Inserting 5 row(s) into table "routes"
[RoutesLoader] The data was successfully loaded into the database
[FrequenciesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[FrequenciesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[FrequenciesTableInterpreter] Matching header with provided column names
[FrequenciesTableInterpreter] Validating 11 row(s) according to the column types
[FrequenciesTableInterpreter] Validation completed, the resulting table has 11 row(s) and 4 column(s)
[FrequenciesLoader] Opening database file ./sample-feed.db
[FrequenciesLoader] Dropping previous table "frequencies" if it exists
[FrequenciesLoader] Creating table "frequencies"
[FrequenciesLoader] Inserting 11 row(s) into table "frequencies"
[FrequenciesLoader] The data was successfully loaded into the database
[FareRulesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[FareRulesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[FareRulesTableInterpreter] Matching header with provided column names
[FareRulesTableInterpreter] Validating 4 row(s) according to the column types
[FareRulesTableInterpreter] Validation completed, the resulting table has 4 row(s) and 5 column(s)
[FareRulesLoader] Opening database file ./sample-feed.db
[FareRulesLoader] Dropping previous table "fare_rules" if it exists
[FareRulesLoader] Creating table "fare_rules"
[FareRulesLoader] Inserting 4 row(s) into table "fare_rules"
[FareRulesLoader] The data was successfully loaded into the database
[FareAttributesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[FareAttributesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[FareAttributesTableInterpreter] Matching header with provided column names
[FareAttributesTableInterpreter] Validating 2 row(s) according to the column types
[FareAttributesTableInterpreter] Validation completed, the resulting table has 2 row(s) and 6 column(s)
[FareAttributesLoader] Opening database file ./sample-feed.db
[FareAttributesLoader] Dropping previous table "fare_attributes" if it exists
[FareAttributesLoader] Creating table "fare_attributes"
[FareAttributesLoader] Inserting 2 row(s) into table "fare_attributes"
[FareAttributesLoader] The data was successfully loaded into the database
[CalendarCSVInterpreter] Parsing raw data as CSV using delimiter ","
[CalendarCSVInterpreter] Parsing raw data as CSV-sheet successfull
[CalendarTableInterpreter] Matching header with provided column names
[CalendarTableInterpreter] Validating 2 row(s) according to the column types
[CalendarTableInterpreter] Validation completed, the resulting table has 2 row(s) and 10 column(s)
[CalendarLoader] Opening database file ./sample-feed.db
[CalendarLoader] Dropping previous table "calendar" if it exists
[CalendarLoader] Creating table "calendar"
[CalendarLoader] Inserting 2 row(s) into table "calendar"
[CalendarLoader] The data was successfully loaded into the database
[CalendarDatesCSVInterpreter] Parsing raw data as CSV using delimiter ","
[CalendarDatesCSVInterpreter] Parsing raw data as CSV-sheet successfull
[CalendarDatesTableInterpreter] Matching header with provided column names
[CalendarDatesTableInterpreter] Validating 1 row(s) according to the column types
[CalendarDatesTableInterpreter] Validation completed, the resulting table has 1 row(s) and 3 column(s)
[CalendarDatesLoader] Opening database file ./sample-feed.db
[CalendarDatesLoader] Dropping previous table "calendar_dates" if it exists
[CalendarDatesLoader] Creating table "calendar_dates"
[CalendarDatesLoader] Inserting 1 row(s) into table "calendar_dates"
[CalendarDatesLoader] The data was successfully loaded into the database
[AgencyCSVInterpreter] Parsing raw data as CSV using delimiter ","
[AgencyCSVInterpreter] Parsing raw data as CSV-sheet successfull
[AgencyTableInterpreter] Matching header with provided column names
[AgencyTableInterpreter] Validating 1 row(s) according to the column types
[AgencyTableInterpreter] Validation completed, the resulting table has 1 row(s) and 4 column(s)
[AgencyLoader] Opening database file ./sample-feed.db
[AgencyLoader] Dropping previous table "agency" if it exists
[AgencyLoader] Creating table "agency"
[AgencyLoader] Inserting 1 row(s) into table "agency"
[AgencyLoader] The data was successfully loaded into the database

(Hint: Omitting rows on StopTimes is correct behavior, because the schema in the google-sample-feed is violated)

Schema of SQLite Sink after executing the pipeline

type	name	tbl_name	rootpage	sql
table	trips	trips	11	CREATE TABLE "trips" ("route_id" text,"service_id" text,"trip_id" text,"trip_headsign" text,"direction_id" text,"block_id" text,"shape_id" text)
table	stops	stops	2	CREATE TABLE "stops" ("stop_id" text,"stop_name" text,"stop_desc" text,"stop_lat" text,"stop_lon" text,"zone_id" text,"stop_url" text)
table	stop_times	stop_times	3	CREATE TABLE "stop_times" ("trip_id" text,"arrival_time" text,"departure_time" text,"stop_id" text,"stop_sequence" text,"stop_headsign" text,"pickup_type" text,"drop_off_time" text,"shape_dist_traveled" text)
table	shapes	shapes	4	CREATE TABLE "shapes" ("shape_id" text,"shape_pt_lat" text,"shape_pt_lon" text,"shape_pt_sequence" text,"shape_dist_traveled" text)
table	routes	routes	5	CREATE TABLE "routes" ("route_id" text,"agency_id" text,"route_short_name" text,"route_long_name" text,"route_desc" text,"route_type" text,"route_url" text,"route_color" text,"route_text_color" text)
table	frequencies	frequencies	12	CREATE TABLE "frequencies" ("trip_id" text,"start_time" text,"end_time" text,"headway_secs" text)
table	fare_rules	fare_rules	6	CREATE TABLE "fare_rules" ("fare_id" text,"route_id" text,"origin_id" text,"destination_id" text,"contains_id" text)
table	fare_attributes	fare_attributes	7	CREATE TABLE "fare_attributes" ("fare_id" text,"price" text,"currency_type" text,"payment_method" text,"transfers" text,"transfer_duration" text)
table	calendar	calendar	8	CREATE TABLE "calendar" ("service_id" text,"monday" text,"tuesday" text,"wednesday" text,"thursday" text,"friday" text,"saturday" text,"sunday" text,"start_date" text,"end_date" text)
table	calendar_dates	calendar_dates	9	CREATE TABLE "calendar_dates" ("service_id" text,"date" text,"exception_type" text)
table	agency	agency	10	CREATE TABLE "agency" ("agency_id" text,"agency_name" text,"agency_url" text,"agency_timezone" text)

[rhazn]

Cool af 🥳

[georg-schwarz]

Should we move the mobility.jv to the /examples directory @rhazn?

[rhazn]

Yeah, sorry I did not even check where it is. But makes sense to move into the examples directory. We can embed the original file into the RFC.