CTIS standard errors should incorporate survey weights

[capnrefsmmat]

Actual behavior

The standard errors for binary proportion signals (i.e. most of them) do not use weights.

Expected behavior

The standard errors should incorporate the weights, as documented.

Constraints

• If we simply change the SE code, SEs issued before the changeover date would be unweighted, and those afterwards would be weighted: very confusing
• We want to retain the issue history of our signals
• Regenerating the issue history of fb-survey from scratch would be prohibitively time-intensive, both computationally and for engineering to figure out precisely how to do it

Proposal

1. Update the code to produce weighted SEs
2. For old SEs in our API's database, serve them to clients with the SE labeled "unweighted SE" or some similar name; make sure the documentation explains the difference. This would require an update to the API server in delphi-epidata
3. To the extent possible, backfill past dates with the new SEs (as a new issue).

Suppose day S is the day we switch to producing weighted SEs. Then:

• All estimates with an issue date on or after S would have weighted SEs
• All estimates with an issue date before S would have unweighted SEs
• But for time_values before S, we would create a new issue on S that contains the same estimate and corrected standard error

Hence the only people who would see unweighted SEs would be people who use the as_of argument to our API; anyone simply asking for estimates for a specific date would get the weighted SE.

Open questions

1. Is it plausible to report old SEs with a different column name? This would require some code changes to the API server; @krivard would such source-specific hacks be welcome in the code?
2. How long will it take to rewrite the standard error code and test it?
3. How long would it take to produce new estimates for all past time_values, and would that be an issue for our database? I'm guessing multiple days just of code runtime; not sure about database ingestion of a new issue for every geo_value and time_value and signal; @krivard would this be prohibitively database-killing?

[krivard]

Is it plausible to report old SEs with a different column name? This would require some code changes to the API server; @krivard would such source-specific hacks be welcome in the code?

I'm less worried about the code changes than I am about the fact that we'd need to store the old SEs somewhere. Adding an extra column to the database for this is strongly discouraged. You could however do it as a big pile of alternately-named signals, though it would ~double the storage footprint of the survey. Back in February the survey was only 3% of the covidcast table and I'd have said 2x-ing that was no big deal, but there've been a lot of changes since then so I'd want to see updated numbers. Let me know if you're still thinking about this and I can have Joe run them up.

not sure about database ingestion of a new issue for every geo_value and time_value and signal; @krivard would this be prohibitively database-killing?

Not database-killing, or even Automation-killing if we're smart about it. The basic batch issue upload functionality is best suited to a half dozen issues at a time or less, but I've written a utility to convert files in batch issue format to SQL LOAD DATA INFILE format which should be much quicker (we did 1.5B rows in 4 days, and 160M rows in ~1h for bigint). Joe's double-checking that it doesn't do anything surprising but once I get his okay we should be set.

[capnrefsmmat]

We discussed this at the survey meeting today and agreed that, if keeping a second column is too much complexity, we're fine with removing the standard errors on old issue dates. Basically:

1. Generate a new issue with correct standard errors, covering all signals over all time. Feed this into your batch upload thing.
2. UPDATE covidcast SET standard_error = NULL WHERE issue < some_date

Facebook isn't aware of anyone who needs standard errors in past issue dates, and neither are we.

(For anyone who really wants standard errors on old issue dates, in principle they can be approximated: we can calculate the "design effect", which is the ratio between weighted and unweighted SEs, and publish it monthly or something. Then anyone can get an old issue, do the binomial standard error formula, and multiply by our design effect. It won't be exact because the exact correction depends on the exact survey respondents included in the calculation, but it'll be pretty good.)

[krivard]

Sounds good -- we won't even need to do a batch upload for a single issue like that; it can just go into the normal daily pipeline.