-
-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Continuously get the latest cases from the iquery page #4081
Comments
Some notes from our talk today about this:
Did I miss something? |
I had a different idea about jitter, but I like yours better.
It'll also be ignored if created by a lower hit by the probe, like if the probe has a hit on 8 and 16, but misses on 32, you'd save the 8 and 16, but only trigger the signal with the 16, not the 8.
The only other detail is that the daemon has a loop cycle that it uses to schedule probes. Currently we're imagining a five minute cycle, but a configuration for it is probably a good idea. We'll also want to avoid probing for five minutes in a court where we just did a sweep. |
Oh, I see. I thought we were going to save only the latest hit when probing (16 in this case) so only a single signal is triggered. But I think it's a good idea to save all the hits found by the probe and do something to only trigger the signal on the latest one.
Got it. Yeah, here we can set a timestamp for the court in Redis when scheduling the tasks for the sweep. Do you think it's enough to wait 5 minutes since the latest sweep task for the court was scheduled, or will we need to figure out somehow when all the tasks for the court have been completed and start the countdown from there? |
Sure, or it can happen in the task. I think that's probably better, so the save completes quickly.
I think so, yeah. If we're doing this correctly, we shouldn't have more than 256 items being scraped at a given time, and five minutes should be enough time to do that. |
As we move towards really great alerts, we need to make sure we have every case as it comes out. The way to do this (freely and quickly) is to scrape the iquery page.
A few notes:
This is hard because some cases are sealed and others don't exist yet.
It might be better to use the Hidden API to check docket numbers, and then use the result from that to scrape iquery. Somehow we need to figure out what is sealed and what is not; this might be one way?
In a perfect world, we would know about new cases instantly, but if we can know in:
We'll want this as a daemon that runs around the clock.
We also have a client currently that wants this done, so it's a great time to work on this!
The text was updated successfully, but these errors were encountered: