[FEATURE] Mobility Extension (RFC0002)

[schlingling]

User Story

1. As a {user}
2. I want {to archive GTFS-data from an http-endpoint}
3. So that {this kind of domain specific data gets stored in an SQLite-Database according to the GTFS relational model}

Notes

A basis for all UACs is the corresponding RFC0002 Mobility extension. The RFC got qualified by multiple iterations, which are listed below. One UACs represents a single requirement, extraced from the final, accepted iteration.
Iterations of RFC0002 Mobility extension:

Iteration	Scope	PR
1	RFC mobility extension using collections: inital concept	#111
2	FC mobility extension using collections: refinement after feedback-loop	#115
3	RFC mobility extension using file-pickers: change of concept	#116
4	RFC mobility extension using file-pickers: refinement after feedback-loop	#117
5	RFC mobility extension using file-pickers: RFC-status ACCEPTED	#119

User Acceptance Criteria

• [UAC-1] New io-datatype File is implemented. Via New io-datatype File (UAC-1 of #123) #125 and Implemented composite pattern for filesystem #256
• [UAC-2] New io-datatype FileSystem is implemented. Via New io-datatype FileSystem, None, Folderstructure (UAC-2-3-8 of #123) #126 and Implemented composite pattern for filesystem #256
• [UAC-3] New io-datatype None is implemented. Via New io-datatype FileSystem, None, Folderstructure (UAC-2-3-8 of #123) #126
• [UAC-4] The io-datatype Table stores the table's name. Obsolet via [UAC-5] DISCUSSION The block LayoutValidator process the new table name coming from Table. #164 and [UAC-4] The io-datatype Table stores the table's name (UAC-4 of #123) #165
• [UAC-5] The block LayoutValidator process the new table name coming from Table. Obsolet via [UAC-5] DISCUSSION The block LayoutValidator process the new table name coming from Table. #164
• [UAC-6] The example.jv-files holds the tables name in Table-Block. Obsolet via [UAC-5] DISCUSSION The block LayoutValidator process the new table name coming from Table. #164
• [UAC-6.1] The example.jv-files holds the tables name in Loaderblocks. via Refactor mobility.jv to tableInterpreter #166
• [UAC-7] If a precessor of a block outputs None, the execution aborts. via New blocktype FilePicker and abort mechanism (UAC-7-11 of #123) #136
• [UAC-8] Folderstructure for io-datatypes is introduced. Via Folderstructure for io-types (UAC-8 of #123) #124 and via New io-datatype FileSystem, None, Folderstructure (UAC-2-3-8 of #123) #126 and via Implemented composite pattern for filesystem #256
• [ UAC-09] New blocktype HttpExtractor is implemented in std-extension. via New blocktype HttpExtractor (UAC-09 of #123) #134
• [UAC-10] New blocktype ArchiveInterpreter is implemented in std-extension. via New blocktype ArchiveInterpreter (UAC-10 of #123) #135
• [UAC-11] New blocktype FilePicker is implemented in std-extension. via New blocktype FilePicker and abort mechanism (UAC-7-11 of #123) #136
• [UAC-12] The current blocktype CSVFileExtractor is refactored to an CSVInterpreter. via [UAC-12] The current blocktype CSVFileExtractor is refactored to an CSVInterpreter. #168
• [UAC-13] The former extractor-functionality of CSVFileExtractor is covered by the new HTTPExtractor and ArchiveInterpreter. via [UAC-13] The former extractor-functionality of CSVFileExtractor is covered by the new HTTPExtractor and ArchiveInterpreter #169
• [UAC-14] The example.jv-files are adapted to using the new blocks HTTPExtractor and ArchiveInterpreter. via [UAC-13] The former extractor-functionality of CSVFileExtractor is covered by the new HTTPExtractor and ArchiveInterpreter #169 and TODO
• [UAC-15] All conditional columns of an GTFS-Layout are considered as required --> out of scope, via Unified cell range model for CSV Layouts #85
• [UAC-16] The current block SQLiteSink accepts multiple inputs. --> for a PoC, multiple sinks are used, rather than multiple inputs for one sink, this works out of the box
• [UAC-17] The current block SQLiteSink process the new tables' name. obsolet via [UAC-5] DISCUSSION The block LayoutValidator process the new table name coming from Table. #164
• [UAC-18] The current block SQLiteSink does not recreate an DB each call. --> already implemented, if an database exists, the db gets opened, otherwise created.
• [UAC-19] Parallel processing of independ blocks does not interfer the overall execution .--> already implemented
• [UAC-20] Jayvee processes successfully 0002-mobility.jv via [UAC-20] Jayvee processes successfully mobility.jv #180

Examples

A detailled explanation of all UACs and further context is provided by the RFC0002

Definitions of Done

• All PRs has been opened and accepted
• All user acceptance criteria are met
• All tests are passing

[schlingling]

@rhazn May you have a short look, if this issue fits your expectation? Then i would start the implementation.

[rhazn]

Sounds good 👍 , also FYI @felix-oq

[schlingling]

Successfully extended Jayvee by GTFS-Support as proposed in RFC0002. See #180 for an detailed perspective on the result. RFC0002 is therefore fully implemented.

FYI @georg-schwarz @felix-oq @rhazn