Prevalence runs failing #1654
Comments
|
If anyone thinks they could take a crack on figuring out some Python code and would like to pair on fixing this, please reach out, or just book some time with me. There's some information on setting up local development in our README to get started. |
|
Eyeballing it, it looks like the world vaccine data (https://raw.githubusercontent.com/govex/COVID-19/master/data_tables/vaccine_data/global_data/time_series_covid19_vaccine_global.csv) has not been updated since the 21st. For issues like this is it typical to followup with the source to figure out why it hasn't been updated? (changes in release cadence, no longer maintained, etc). Or moreso just make the script handle the lack of data and move on? |
|
Yeah, I've made a practice of following up with the upstream source when things like this happen, which has been pretty effective in general. Note that population vaccination numbers currently don't affect the risk values in the model much, because it's easy to catch and spread Omicron even when you're vaccinated. Given that, I wouldn't have a problem making changes to the safety check, especially if it's done in a way that balances the risk of things failing silently as a result. We've encountered data feeds being retired, data formats radically changing, upstream providers having issues they don't fix until we talk to them, etc...having something to tell us about those is useful. Ideally we'd have a low-noise way to publish warnings about things like this without failing prevalence entirely. We don't today - the Sentry references in the code aren't configured to go to an account I have access to. I've thought about adding a Sentry Slack integration or even just a direct Slack integration from the Python script, so we can at least get those piped into places that active contributors can see. |
|
Left a github issue comment for the upstream source but given the state of the other issues I'm not expecting a near term response 🤞 Gotcha, makes sense. Sentry does allow open source projects to apply for a free account via https://sentry.io/for/open-source if a new account for current contributors is needed. I'm assuming the slack you refer to is an instance used for contributors? |
Thanks for filing that! They don't provide source code or logs for their data ingestion pipeline. That said, I notice in their README they list three sources for the upstream data:
Wonder if there's an obvious point where things are stuck upstream.
Right on. Be aware that their free plan does have a 50k monthly error limit, which I suspect we'd blow through with the current configuration on what gets logged. Maybe the open source plan has a higher limit... Yeah, we have a Slack instance we can use - I can get you access if you like. It's a ghost town in terms of actual discussion, but may be useful for integrations like this to post into. |
|
I'm happy to take a look. I'm not a python expert, but I know my way around well enough. I only took a cursory look, but I didn't see any try/catch action going on. @apiology I'm going to grab some time on calendly with you for later this week to get better aquatinted with the code. I also know actions FWIW. |
|
Fixed upstream--thanks to @shawnbiesan2 for alerting folks! |
Our daily prevalence updates have failed the last four days (11/23-11/26). This is an example log, which is unfortunately hidden to folks who aren't part of the org:
The text was updated successfully, but these errors were encountered: