-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Where are the Metro logs? #55
Comments
Potentially related to opencivicdata/scrapers-us-municipal#328. |
I unzipped the most recent log archive:
The last Metro scrape we have logs for ran on 7/18:
I'm not certain, but when I was spelunking re: opencivicdata/scrapers-us-municipal#328, I checked the max updated date on bills in the Councilmatic database, and it may have been 7/18. Are the scrapes not actually running? We break the Metro scrapes into scrape and import steps, so I would at least expect to see logs for the scrapes, even if the imports fail. Very curious... |
We last deployed the scrapers at 9 a.m. 7/17, which is 2 p.m. UTC time. Per the logs, scrapes continued to run through Saturday, 7/18, before "stopping" (or not being logged). So I'm not sure that this was caused by a change to the deployed code. |
FWIW, the last scrape coincides with the Saturday windowed scraping schedule: scrapers-us-municipal/scripts/la-metro-crontask Lines 67 to 74 in 27df297
It's as if scheduling for the new week never kicked in? |
This would seem to be the scrape where a huge number of bills were updated:
|
Confirming that both of the public bills Shelly provided as examples had inaccessible gateway links during the implicated scrape:
|
Hm, it seems like there are a bunch of Metro scrapes just chilling on the OCD server:
This would explain why there are no logs – old file locks are preventing the new scrapes from running. How did this happen? And is it safe to terminate them? |
We use However! It looks like the full scrape does not have this option: scrapers-us-municipal/scripts/la-metro-crontask Lines 16 to 18 in 27df297
So, that might explain the build up of full scrapes. But why are the windowed scrapes hanging out??? More on |
Oh, there's just the one fast full event scrape causing the bottleneck... So why is that hung???? |
Decided that it wasn't worth the time investment to investigate the bottleneck since we are moving away from this pattern for Metro. Instead, I killed each of the blocking processes with Hark! The next regularly scheduled event scrape ran and logged as expected. |
I'm trying to debug some strange behavior in the Metro scraper. I'm noticing that the main log (
/tmp/lametro.log
) is empty, and the log before (/tmp/lametro.log.1
) only has logs from the scrape I ran manually, and nothing else.I searched the syslog to confirm scrapes were running, and it seems like they are, and that they're configured to append logs to the expected log.
So why don't we have any logs????
Importantly, both the Chicago and NYC scrapers seem to be updating their respective logs as expected!
The text was updated successfully, but these errors were encountered: