New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: filter to fill in stop times from schedules #243
Merged
Commits on Jun 13, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 37f5c23 - Browse repository at this point
Copy the full SHA 37f5c23View commit details
Commits on Jun 14, 2022
-
refactor: move
GTFS
out ofFilter
namespaceMove `Concentrate.Filter.GTFS.*` to `Concentrate.GTFS.*`. We are introducing a `Concentrate.GroupFilter` that will use `GTFS` data, so organizing this part of the app under the `Filter` namespace doesn't make as much sense as it did before.
Configuration menu - View commit details
-
Copy full SHA for b61c59f - Browse repository at this point
Copy the full SHA b61c59fView commit details
Commits on Jun 27, 2022
-
fix: potential periodic
GTFS.PickupDropOff
issueBuilding the records to be inserted into ETS can take a while (over a minute), and since all records are deleted from the table before this work begins, the module will return `:unknown` for all lookups during this time. This issue would occur once per hour, as per the static GTFS refresh interval, and would result in `RemoveUnneededTimes` temporarily not working. This refactor applies the same approach used in `StopIDs`: build all records up-front, then clear the table and immediately insert all the records in one batch. This still takes a second or two, but the time gap is much shorter, reducing the chance that an incorrect feed would be produced and the time until a corrected feed would be produced. Eliminating the issue entirely will require a more complex approach, such as writing some "upsert" logic which also deletes records that shouldn't be in the table anymore, or using Mnesia for its transaction support.
Configuration menu - View commit details
-
Copy full SHA for 666cc90 - Browse repository at this point
Copy the full SHA 666cc90View commit details -
Configuration menu - View commit details
-
Copy full SHA for c3ca297 - Browse repository at this point
Copy the full SHA c3ca297View commit details
Commits on Jun 29, 2022
-
refactor: combine stop-time-related GTFS servers
Having three different servers loading `stop_times.txt` and maintaining their own subset of it was resulting in very high CPU and memory usage. Combining these, similar to `GTFS.Trips`, saves on system resources and significantly reduces the "warm-up time" in which filters are not fully functional. Approximate numbers on a developer machine: Metric | pre `StopTimes` | post `StopTimes` | consolidated ------------ | --------------- | ---------------- | ------------ Update Time | 1.5 min | 4 min | 1.5 min Peak Memory | 2.2GB | 3.3GB | 2.0GB Steady-State | 1.3GB | 1.7GB | 880MB The first column is from when there were only two `stop_times.txt` servers, `PickupDropOff` and `StopIDs`. We can see that consolidation is also an improvement over these numbers. There are some incidental changes that should not have any effect on the behavior of the app: * `RemoveUnneededTimes` no longer works with stop time updates that are missing a `stop_sequence`. This should not have happened anyway since ac72186, which formalized the requirement that all stop time updates should have a stop sequence for merging to work correctly. * `RemoveUnneededTimes` and `IncludeStopID` no longer work with GTFS feeds that have _only_ a `stop_times.txt`. Like `ScheduledStopTimes`, they now require the full chain of `stop_times.txt`, `trips.txt`, `routes.txt`, and `agencies.txt` to be present and valid, though the files are unused in these filters. Both the old and new behaviors are accidents of the implementation details, and the new behaviour could be changed if we wanted to explicitly support invalid "partial" GTFS feeds, but currently we have no need for this.
Configuration menu - View commit details
-
Copy full SHA for edbc809 - Browse repository at this point
Copy the full SHA edbc809View commit details
Commits on Jul 19, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 3b1f93e - Browse repository at this point
Copy the full SHA 3b1f93eView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.