feat(state,parse): Perform partial updates on StopEvent - #1065
Merged
Conversation
runkelcorey
marked this pull request as ready for review
August 6, 2026 18:51
runkelcorey
requested review from
khansen-mbta and
lemald
and removed request for
a team
August 6, 2026 18:51
lemald
reviewed
Aug 6, 2026
lemald
reviewed
Aug 6, 2026
lemald
reviewed
Aug 6, 2026
lemald
reviewed
Aug 6, 2026
lemald
reviewed
Aug 6, 2026
lemald
reviewed
Aug 6, 2026
Co-authored-by: Eddie Maldonado <eddie@lemald.org>
lemald
approved these changes
Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Asana Ticket: 🐞 State.StopEvent times out on refresh
What changes does this PR propose?
Changes refresh strategy for
StopEventfrom a full load to an incremental update by:handle_new_statecallback that filters incoming stop event JSONs for records that are newer than a configurable timestamp (defaulting to 2 hours, the same that Flashback aims for)post_commit_hookcallback that evicts records from the mnesia table if they are older than the configurable timestampSince
StopEventis a:settable, incoming values with the same keys will overwrite existing values.How were these changes validated?
State.StopEventandParse.StopEventthat different cases of partial updatesdev-blueanddev-green(which is running the current, full-load strategy):dev-greendev-blueThese are crazy improvements so I feel like I'm missing a logging statement to capture the full overhead of the partial load. However, we should expect something like this: of the 116,740 events in the stop events dataset I downloaded yesterday, only 356 (0.3%) were updated or inserted the last 30 seconds of the dataset.
As a sanity check, I also counted the number of times each of these function were executed: since this is an event loop dictated by the time it takes to refresh data + sleep time, the more times something is invoked can indicate performance impacts.
dev-bluecompleted 1,059 refreshes ofStopEventin 30 minutes, which translates to once per 1.7s, whiledev-greencompleted 432 refreshes in the same time period, a rate of one refresh every 4s.Performance-wise, these changes seem very impactful.
What questions should reviewers consider?
timestampbe a datetime or is it ok to have as a Unix timestamp?