Skip to content

Refactor speeds pipeline3 #1009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 31, 2024
Merged

Refactor speeds pipeline3 #1009

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7d24d3c
backfill all the dates with new workflow
Jan 30, 2024
90e13cf
add segment geom for multiday avg
Jan 30, 2024
52be517
remove unused code
Jan 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions gtfs_funnel/logs/vp_preprocessing.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
2024-01-25 12:58:02.895 | INFO | __main__:<module>:186 - 2023-12-13: condense vp for trip-direction 0:06:17.127762
2024-01-25 13:09:11.256 | INFO | __main__:<module>:194 - 2023-12-13: prepare vp to use in nearest neighbor: 0:11:08.360926
2024-01-25 13:15:43.377 | INFO | __main__:<module>:186 - 2023-11-15: condense vp for trip-direction 0:06:32.119567
2024-01-25 13:27:16.897 | INFO | __main__:<module>:194 - 2023-11-15: prepare vp to use in nearest neighbor: 0:11:33.520352
2024-01-25 13:33:47.613 | INFO | __main__:<module>:186 - 2023-10-11: condense vp for trip-direction 0:06:30.714860
2024-01-25 13:45:08.728 | INFO | __main__:<module>:194 - 2023-10-11: prepare vp to use in nearest neighbor: 0:11:21.115081
2024-01-25 13:51:28.299 | INFO | __main__:<module>:186 - 2023-09-13: condense vp for trip-direction 0:06:19.569858
2024-01-25 14:20:34.069 | INFO | __main__:<module>:143 - 2023-09-13: prepare vp to use in nearest neighbor: 0:10:58.447822
2024-01-27 10:04:12.447 | INFO | __main__:<module>:136 - 2023-12-13: condense vp for trip-direction 0:07:09.337844
2024-01-27 13:09:21.616 | INFO | __main__:<module>:144 - 2023-12-13: prepare vp to use in nearest neighbor: 0:11:47.446912
2024-01-27 14:27:17.221 | INFO | __main__:<module>:136 - 2023-11-15: condense vp for trip-direction 0:07:23.319808
2024-01-27 14:39:49.578 | INFO | __main__:<module>:144 - 2023-11-15: prepare vp to use in nearest neighbor: 0:12:32.357147
2024-01-27 14:46:57.364 | INFO | __main__:<module>:136 - 2023-10-11: condense vp for trip-direction 0:07:07.785109
2024-01-27 14:58:58.072 | INFO | __main__:<module>:144 - 2023-10-11: prepare vp to use in nearest neighbor: 0:12:00.707961
3 changes: 2 additions & 1 deletion gtfs_funnel/vp_condenser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ def condense_vp_to_linestring(
vp_transform.condense_point_geom_to_line,
group_cols = ["trip_instance_key"],
geom_col = "geometry",
other_cols = ["vp_idx", "location_timestamp_local", "vp_primary_direction"],
other_cols = ["vp_idx", "location_timestamp_local",
"vp_primary_direction"],
meta = {
"trip_instance_key": "object",
"geometry": "geometry",
Expand Down
67 changes: 11 additions & 56 deletions rt_segment_speeds/logs/avg_speeds.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,61 +1,16 @@
2024-01-12 18:06:39.922 | INFO | __main__:single_day_averages:161 - shape seg avg 0:03:02.810870
2024-01-12 18:08:46.477 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:02:06.554774
2024-01-12 18:08:52.454 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.976923
2024-01-12 18:08:55.443 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:02.988727
2024-01-12 18:08:55.535 | INFO | __main__:<module>:226 - average rollups for 2023-11-15: 0:05:30.589840
2024-01-12 18:12:19.679 | INFO | __main__:single_day_averages:161 - shape seg avg 0:03:12.836248
2024-01-12 18:14:24.263 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:02:04.583881
2024-01-12 18:14:30.117 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.854436
2024-01-12 18:14:33.483 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:03.366009
2024-01-12 18:14:33.561 | INFO | __main__:<module>:226 - average rollups for 2023-10-11: 0:05:38.025964
2024-01-12 18:17:38.177 | INFO | __main__:single_day_averages:161 - shape seg avg 0:02:53.331595
2024-01-12 18:19:37.364 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:01:59.186956
2024-01-12 18:19:42.466 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.102817
2024-01-12 18:19:45.241 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:02.774644
2024-01-12 18:19:45.320 | INFO | __main__:<module>:226 - average rollups for 2023-09-13: 0:05:11.758315
2024-01-12 18:23:03.853 | INFO | __main__:single_day_averages:161 - shape seg avg 0:03:07.330519
2024-01-12 18:25:05.827 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:02:01.974740
2024-01-12 18:25:11.688 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.860574
2024-01-12 18:25:14.962 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:03.274055
2024-01-12 18:25:15.059 | INFO | __main__:<module>:226 - average rollups for 2023-08-15: 0:05:29.737775
2024-01-12 18:28:02.563 | INFO | __main__:single_day_averages:161 - shape seg avg 0:02:35.697961
2024-01-12 18:29:46.221 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:01:43.657744
2024-01-12 18:29:51.370 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.149411
2024-01-12 18:29:54.318 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:02.947539
2024-01-12 18:29:54.406 | INFO | __main__:<module>:226 - average rollups for 2023-07-12: 0:04:39.346597
2024-01-12 18:32:47.926 | INFO | __main__:single_day_averages:161 - shape seg avg 0:02:41.740822
2024-01-12 18:34:30.637 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:01:42.711175
2024-01-12 18:34:35.408 | INFO | __main__:single_day_averages:188 - trip avg 0:00:04.771437
2024-01-12 18:34:37.957 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:02.549076
2024-01-12 18:34:38.052 | INFO | __main__:<module>:226 - average rollups for 2023-06-14: 0:04:43.645367
2024-01-12 18:37:36.391 | INFO | __main__:single_day_averages:161 - shape seg avg 0:02:47.454496
2024-01-12 18:39:30.098 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:01:53.707229
2024-01-12 18:39:35.127 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.029718
2024-01-12 18:39:37.900 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:02.772840
2024-01-12 18:39:37.970 | INFO | __main__:<module>:226 - average rollups for 2023-05-17: 0:04:59.916877
2024-01-12 18:42:48.234 | INFO | __main__:single_day_averages:161 - shape seg avg 0:03:00.306669
2024-01-12 18:44:44.898 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:01:56.664023
2024-01-12 18:44:50.122 | INFO | __main__:single_day_averages:188 - trip avg 0:00:05.223725
2024-01-12 18:44:53.286 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:03.163784
2024-01-12 18:44:53.356 | INFO | __main__:<module>:226 - average rollups for 2023-04-12: 0:05:15.385673
2024-01-12 18:47:46.971 | INFO | __main__:single_day_averages:161 - shape seg avg 0:02:42.806424
2024-01-12 18:49:40.554 | INFO | __main__:single_day_averages:174 - route dir seg avg 0:01:53.582921
2024-01-12 18:49:45.551 | INFO | __main__:single_day_averages:188 - trip avg 0:00:04.997039
2024-01-12 18:49:48.333 | INFO | __main__:single_day_averages:200 - route dir avg: 0:00:02.782500
2024-01-12 18:49:48.405 | INFO | __main__:<module>:226 - average rollups for 2023-03-15: 0:04:55.047935
2024-01-12 21:40:57.958 | INFO | __main__:multi_day_averages:272 - route seg avg 0:04:51.371722
2024-01-12 21:42:02.380 | INFO | __main__:multi_day_averages:292 - route dir avg 0:01:04.422529
2024-01-12 21:42:02.381 | INFO | __main__:<module>:322 - average rollups for ['2023-10-09', '2023-10-10', '2023-10-11', '2023-10-12', '2023-10-13', '2023-10-14', '2023-10-15']: 0:05:55.797342
2024-01-12 21:58:24.392 | INFO | __main__:multi_day_averages:272 - route seg avg 0:05:11.243450
2024-01-12 21:59:19.820 | INFO | __main__:multi_day_averages:292 - route dir avg 0:00:55.428205
2024-01-12 21:59:19.821 | INFO | __main__:<module>:322 - average rollups for ['2023-04-10', '2023-04-11', '2023-04-12', '2023-04-13', '2023-04-14', '2023-04-15', '2023-04-16']: 0:06:06.674622
2024-01-26 09:07:07.005 | INFO | __main__:single_day_averages:188 - shape seg avg 0:00:50.189845
2024-01-26 09:07:53.785 | INFO | __main__:single_day_averages:221 - route dir seg avg 0:00:46.780283
2024-01-26 09:07:54.887 | INFO | __main__:single_day_averages:238 - trip avg 0:00:01.101242
2024-01-26 09:08:01.918 | INFO | __main__:single_day_averages:270 - route dir avg: 0:00:07.030940
2024-01-26 09:08:01.923 | INFO | __main__:<module>:359 - average rollups for 2024-01-17: 0:01:48.376378
2024-01-26 11:10:44.570 | INFO | __main__:single_day_averages:188 - shape seg avg 0:00:49.868572
2024-01-26 11:11:30.078 | INFO | __main__:single_day_averages:221 - route dir seg avg 0:00:45.508658
2024-01-26 11:11:31.288 | INFO | __main__:single_day_averages:238 - trip avg 0:00:01.209484
2024-01-26 11:11:39.210 | INFO | __main__:single_day_averages:270 - route dir avg: 0:00:07.922106
2024-01-26 11:11:44.669 | INFO | __main__:<module>:390 - average rollups for 2024-01-17: 0:01:53.284500
2024-01-26 11:11:44.669 | INFO | __main__:<module>:390 - average rollups for 2024-01-17: 0:01:53.2845002024-01-27 21:43:32.075 | INFO | __main__:single_day_averages:188 - shape seg avg 0:00:40.913991
2024-01-27 21:44:10.920 | INFO | __main__:single_day_averages:221 - route dir seg avg 0:00:38.844832
2024-01-27 21:44:12.903 | INFO | __main__:single_day_averages:238 - trip avg 0:00:01.982998
2024-01-27 21:44:21.280 | INFO | __main__:single_day_averages:270 - route dir avg: 0:00:08.377181
2024-01-27 21:44:28.045 | INFO | __main__:<module>:390 - average rollups for 2023-12-13: 0:01:40.228117
2024-01-30 10:19:05.280 | INFO | __main__:multi_day_averages:338 - route seg avg 0:02:47.182451
2024-01-30 10:19:42.290 | INFO | __main__:multi_day_averages:377 - route dir avg 0:00:37.010142
2024-01-30 10:19:42.449 | INFO | __main__:<module>:443 - average rollups for ['2023-04-10', '2023-04-11', '2023-04-12', '2023-04-13', '2023-04-14', '2023-04-15', '2023-04-16']: 0:03:24.353678
2024-01-30 10:22:12.414 | INFO | __main__:multi_day_averages:338 - route seg avg 0:02:29.962531
2024-01-30 10:22:54.131 | INFO | __main__:multi_day_averages:377 - route dir avg 0:00:41.717110
2024-01-30 10:22:54.176 | INFO | __main__:<module>:443 - average rollups for ['2023-10-09', '2023-10-10', '2023-10-11', '2023-10-12', '2023-10-13', '2023-10-14', '2023-10-15']: 0:03:11.725898
12 changes: 1 addition & 11 deletions rt_segment_speeds/logs/cut_stop_segments.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,6 @@
2024-01-17 08:51:37.506 | INFO | __main__:<module>:71 - Analysis date: 2023-12-13
2024-01-17 09:13:47.443 | INFO | __main__:<module>:118 - cut segments for all trips: 0:22:09.920160
2024-01-17 11:26:43.442 | INFO | __main__:<module>:135 - cut segments 2023-11-15: 0:22:29.012832
2024-01-17 11:49:11.170 | INFO | __main__:<module>:135 - cut segments 2023-10-11: 0:22:27.725516
2024-01-23 20:53:12.529 | INFO | __main__:<module>:138 - cut segments 2024-01-17: 0:13:13.064306
2024-01-25 17:55:11.789 | INFO | __main__:<module>:151 - cut segments 2024-01-17: 0:15:20.866136
2024-01-25 18:10:52.489 | INFO | __main__:<module>:151 - cut segments 2023-12-13: 0:15:40.699105
2024-01-25 18:26:27.834 | INFO | __main__:<module>:151 - cut segments 2023-11-15: 0:15:35.343763
2024-01-25 18:41:57.495 | INFO | __main__:<module>:151 - cut segments 2023-10-11: 0:15:29.660398
2024-01-25 18:55:44.922 | INFO | __main__:<module>:151 - cut segments 2023-09-13: 0:13:47.425869
2024-01-25 19:11:01.609 | INFO | __main__:<module>:151 - cut segments 2023-08-15: 0:15:16.686175
2024-01-25 19:24:54.027 | INFO | __main__:<module>:151 - cut segments 2023-07-12: 0:13:52.416916
2024-01-25 19:38:21.517 | INFO | __main__:<module>:151 - cut segments 2023-06-14: 0:13:27.489003
2024-01-25 20:21:31.153 | INFO | __main__:<module>:157 - cut segments 2023-05-17: 0:12:39.779362
2024-01-25 20:35:07.936 | INFO | __main__:<module>:157 - cut segments 2023-04-12: 0:13:36.769604
2024-01-25 20:48:11.716 | INFO | __main__:<module>:157 - cut segments 2023-03-15: 0:13:03.778711
2024-01-25 18:41:57.495 | INFO | __main__:<module>:151 - cut segments 2023-10-11: 0:15:29.660398
Loading