-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider alternative methods of determining whether an entity has been updated #107
Comments
I don't think there are, but I had a potentially useful idea for a windowed scrape (at least for events). Right now, we attempt the widowed scrapes attempt to find all events that have been updated after the beginning of the window. It would probably be helpful if we also scraped events that are scheduled to occur after the beginning of the window. These recent and upcoming events are the ones most likely to change. An increased false positive rate seems worth it if this helps us really reduce the false negatives. |
@fgregg That's really smart! |
Could do something similar with bills by using
and the other date field on matters. |
let me know if you'd like to see a PR |
@fgregg We have the go-ahead from Metro to prioritize this for May. I'd love to review a PR at your convenience. |
There are several known actions that change a bill or event, but do not update the last modified timestamp. These include making an agenda or media link public. This means that those entities will not be captured in so-called "windowed" scrapes (that is, scrapes that only check for entities updated since a certain timestamp). One option for capturing those changes is to run full scrapes more frequently, but that can be a time and resource intensive step. Are there other ways to determine when an entity has been updated?
The text was updated successfully, but these errors were encountered: