Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Resolve San Francisco Pipeline Yaml Variable Assignment Issue #489

Merged
merged 32 commits into from
Oct 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
15ffe7e
feat: san francisco pipelines integrated into one dag.
nlarge-google May 6, 2022
756719c
Feat: Adding municipal calendar pipeline. Not production Ready.
nlarge-google Sep 20, 2022
0ce5fd0
Merge branch 'GoogleCloudPlatform:main' into san-francisco
nlarge-google Sep 20, 2022
478b821
fix: Continuing SF municipal calendar changes.
nlarge-google Sep 20, 2022
c42c995
feat: Added sf_calendar. Is tested successfully.
nlarge-google Sep 21, 2022
f3059ad
fix: Removed cluster create/delete.
nlarge-google Sep 21, 2022
f2bfa3d
feat: Added muni_routes
nlarge-google Sep 22, 2022
122108e
fix: Resolved reference to schema file.
nlarge-google Sep 22, 2022
9174e71
feat: Added SF Municipal Shapes
nlarge-google Sep 23, 2022
e1c5f71
feat: Added SF Municipal Stops
nlarge-google Sep 23, 2022
d379f59
Feat: Added SFPD Incidents.
nlarge-google Sep 24, 2022
e6e89c5
fix: Misc Fixes.
nlarge-google Sep 26, 2022
709665e
fix: Integration Testing.
nlarge-google Sep 26, 2022
219a358
Merge branch 'GoogleCloudPlatform:main' into san-francisco
nlarge-google Sep 26, 2022
0e0b347
fix: Flake etc.
nlarge-google Sep 26, 2022
3bc0796
fix: Misc.
nlarge-google Sep 26, 2022
70d39fe
fix: Misc.
nlarge-google Sep 26, 2022
a099f0e
feat: Includes incomplete version of stop_times. Not production Ready.
nlarge-google Sep 29, 2022
4eff3ca
Merge branch 'main' into san-francisco
nlarge-google Sep 29, 2022
bb4037c
fix: rework regex. Not production Ready.
nlarge-google Sep 30, 2022
4ebff05
fix: Resolve misc issues. Clean up code.
nlarge-google Sep 30, 2022
becee0b
fix: Resolved Flake Issues.
nlarge-google Sep 30, 2022
d196e14
fix: Misc.
nlarge-google Sep 30, 2022
a59d3d8
fix: Fixed incorrect schema reference.
nlarge-google Sep 30, 2022
7b9aa53
fix: Final Test fixes for sf_muni_stop_times.
nlarge-google Sep 30, 2022
7830b31
feat: First parse municipal fares. Not production Ready.
nlarge-google Oct 3, 2022
af5dbd0
fix: Completed local testing of muni fares
nlarge-google Oct 3, 2022
0a40b3c
feat: Incorporated muni fares.
nlarge-google Oct 4, 2022
93c3e16
fix: flake issues.
nlarge-google Oct 4, 2022
b30eef5
fix: Resolved missed change to pipeline yaml variable
nlarge-google Oct 5, 2022
d25c151
Merge branch 'main' into san-francisco
nlarge-google Oct 5, 2022
fcf30b4
Merge branch 'GoogleCloudPlatform:main' into san-francisco
nlarge-google Oct 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions datasets/san_francisco/pipelines/san_francisco/pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -515,8 +515,8 @@ dag:
}
STARTS_WITH_PATTERN_LIST: >-
[
[ ["direction", "direction"], [ "0", "O" ] ],
[ ["direction", "direction"], [ "1", "I" ] ],
[ ["direction_id", "direction_id"], [ "0", "O" ] ],
[ ["direction_id", "direction_id"], [ "1", "I" ] ],
[ ["SERVICE_CA", "SERVICE_CA"], [ "nan", "" ] ]
]
REORDER_HEADERS_LIST: >-
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@
"TARGET_GCS_PATH": "data/san_francisco/transit_municipal_trips/data_output.csv",
"SCHEMA_PATH": "data/san_francisco/schema/sf_muni_trips_schema.json",
"RENAME_HEADERS_LIST": '{\n "DIRECTION": "direction_old",\n "direction_id": "direction",\n "SERVICE_CA": "service_category",\n "shape": "trip_shape"\n}',
"STARTS_WITH_PATTERN_LIST": '[\n [ ["direction", "direction"], [ "0", "O" ] ],\n [ ["direction", "direction"], [ "1", "I" ] ],\n [ ["SERVICE_CA", "SERVICE_CA"], [ "nan", "" ] ]\n]',
"STARTS_WITH_PATTERN_LIST": '[\n [ ["direction_id", "direction_id"], [ "0", "O" ] ],\n [ ["direction_id", "direction_id"], [ "1", "I" ] ],\n [ ["SERVICE_CA", "SERVICE_CA"], [ "nan", "" ] ]\n]',
"REORDER_HEADERS_LIST": '[\n "trip_id",\n "route_id",\n "direction",\n "block_id",\n "service_category",\n "trip_headsign",\n "shape_id",\n "trip_shape"\n]',
},
resources={
Expand Down