Consider alternative methods of determining whether an entity has been updated

[hancush]

There are several known actions that change a bill or event, but do not update the last modified timestamp. These include making an agenda or media link public. This means that those entities will not be captured in so-called "windowed" scrapes (that is, scrapes that only check for entities updated since a certain timestamp). One option for capturing those changes is to run full scrapes more frequently, but that can be a time and resource intensive step. Are there other ways to determine when an entity has been updated?

[fgregg]

I don't think there are, but I had a potentially useful idea for a windowed scrape (at least for events).

Right now, we attempt the widowed scrapes attempt to find all events that have been updated after the beginning of the window.

It would probably be helpful if we also scraped events that are scheduled to occur after the beginning of the window.

These recent and upcoming events are the ones most likely to change.

An increased false positive rate seems worth it if this helps us really reduce the false negatives.

[hancush]

@fgregg That's really smart!

[fgregg]

Could do something similar with bills by using

MatterAgendaDate
MatterIntroducitonDate

and the other date field on matters.

[fgregg]

let me know if you'd like to see a PR

[hancush]

@fgregg We have the go-ahead from Metro to prioritize this for May. I'd love to review a PR at your convenience.