Production for Backfill Correction

[jingjtang]

We already have most of the work done in covid-19(the private repo).
The main goal of this project is to provide backfill correction for the values that are reported everyday.

Work left

• Move the pipeline from covid-19 repo to covidcast-indicators @nmdefries
• Decision on the training frequency and training dates
• Generate historical backfill files for chng, quidel, etc and load to midas. @jingjtang
• Change/not change the path to stored backfill files
• Decision on the way to publish the output @RoniRos @krivard
  • Currently, we have all the corrections stored in csv files
  • Or make them new signals?
  • Or add them as extra columns for the existing signals? (seems impossible?)
  • Or publish the output to an AWS S3 bucket?

2022-10-06 - Engineering meeting notes

[nmdefries]

We could also publish the output to an AWS S3 bucket.

[nmdefries]

Decision on the training frequency and training dates

To start the conversation off on this, my understanding was that we'd train the models once a month on the first of the month at night (when we normally schedule longer-running processes).

[krivard]

• make them new signals
  • pro: easy for any epidata user to incorporate into their existing pipelines
  • con: storage & performance impact on covidcast endpoint
  • con: separate signals obscures relationship to uncorrected values
• add them as extra columns for the existing signals
  • pro: closely associates corrections with uncorrected values
  • con: probably best to store these in a separate table, which would require a significant (3-4 weeks? similar to covid_hosp) development effort on its own, plus whatever handling in the covidcast endpoint to conditionally glue it on
  • con: probably makes output incompatible with existing API clients. we could adapt the clients to handle it but then anyone who doesn't update gets a crash.
• publish the output to an AWS S3 bucket
  • pro: uploads are easy
  • con: separate access from all other epidata. more work for users

[RoniRos]

We need to establish structured relationships between different signals, e.g. those related by JIT work. Backfill projection signals should be part of that.
I will add this task to our task list.
We need to decide the priority of such a task, and based on that, whether backfill-projection should wait for it.
If it shouldn't (which is what I expect we might decide), we should use a temporary solution.

[krivard]

Proposed AWS S3 file organization:

Unique file per

• source
• signal
• region
• correction computation date

Then a single-region timeseries is a single file fetch.

Storage for this is going to get big pretty quickly, since we're saving the full correction history. To save space, consider:

• gzip parquet format
• limited precision
• excluding replicated data

[nmdefries]

excluding replicated data

Katie, can you expand on this? Are you talking about diffing the previous day's and the current day's data to find changes only?

The compression and limiting precision points make sense.

[RoniRos]

The structure we are designing here for storing backfill projections is pretty much the same as that needed for storing forecasts, nowcasts, and backcasts. In fact backfill projection is essentially a backcast of a particular indicator. So I'd like to bring @ryantibs , @brookslogan and potentially other forecast-related folks into this discussion.

I think the relevant dimensions are:

• Source+Signal
• geo-location (aka "region")
• reference date
• as-of datetime (better than "computation date" because it could be computed retrospectively, as long as it uses the covariates as of that date)
• quantile

To put these in S3 files, I agree it make sense to separate the data at least by:

• Source+Signal
• geo-location

and possibly also by:

• quantile (because users may be interested is specific subsets of quantiles, like 50%, or (10%, 50%, 90%)).

This leaves a 2D table of {reference_date X as-of_datetime}. Since these data will be produced every as_of day, it makes sense to add as_of to the file name/identify, and make the file consist of ~60 values corresponding to lags from 0 or 1 up to ~60, corresponding to reverse-successive reference dates. Since we are going to have all ~60 values, we don't need to store the actual lag (and we can always represent a missing value with an extra separator).

Alternatively, we could store all the quantiles in the same file. But it can get messy if different quantile sets are produced at different times or for different indicators.

@ryantibs , @brookslogan : how are quantile forecasts stored by us? By the Forecast Hub?

[krivard]

make the file consist of ~60 values corresponding to lags from 0 or 1 up to ~60, corresponding to reverse-successive reference dates

does that mean we're okay with users needing to pull multiple files in order to build a time series covering the whole pandemic?

[krivard]

excluding replicated data

Katie, can you expand on this?

I do not mean diffing; that would be a last resort as it would require much more complicated data fetching capabilities than are easy to do in S3. It's possible, and we'd have help setting it up (it was recommended by one of the Amazon data teams), but it would take substantial effort.

I mean pulling any data that's the same for all rows of the file out into the filename or some kind of header, and pulling any categorical data with long human-readable names into an index file and referring to it by a numeric id instead.

[krivard]

as-of datetime (better than "computation date" because it could be computed retrospectively, as long as it uses the covariates as of that date)

this will create additional operational costs when source (covariate) data patches are applied, and the operational cost of data patches is already high. i'll follow up with you offline.

[brookslogan]

Trying to give some quick answers and think more about this later.

We currently store forecasts in a couple of formats, with no real care for saving space:

• One RDS for each internal forecaster in our system. (Stored in S3 cache.)
• One CSV for each submitted forecast.
• One RDS for all forecasts. Obviously not scalable, but at one point it worked for all our pseudoprospective forecaster candidates.

The Hub, at least in their GitHub, stores:

• One CSV per forecast. Originally contained in the repo contents, now backed via Git LFS.
• Not sure what they do in Zoltar.

[RoniRos]

make the file consist of ~60 values corresponding to lags from 0 or 1 up to ~60, corresponding to reverse-successive reference dates

does that mean we're okay with users needing to pull multiple files in order to build a time series covering the whole pandemic?

Possibly. Backfill corrections don't matter beyond a 60 day lag, and for any reference_date older than 60 days ago, they are not needed except for retrospective error analysis and training forecasting models.

More generally, here are the use cases I can think of for backfill-projected signals:

1. Displaying, in real-time, the most uptodate estimates available of the finalized values of the recent past: for a fixed signal, region, quantile and as-of_datetime, get a list of 60 projected values, where reference_date goes back from as-of_date to as-of_date-60, and in parallel lag goes up from 0 to 60. This would be the most common operational use of this signal. It will also be the most likely set of covariates for forecasting. In my proposal above, it will require a single file.

2. Showing a map of the signal value in, e.g. 50 U.S. states, using not the reported value but rather the most recently estimated 50% backfill-projected value. In my proposal above, this will require 50 files (or 3000 if we do counties). If you want to step the map forward or backward in reference_date, each such step will take another 50 (or 3000) files. Not great.

3. Demonstrating the accuracy of backfill projection: for a fixed signal, region, historical reference period (>60 days ago), and handful of quantiles, plot the (relative) error between the finalized value and the projected value as function of the lag. This requires 2D data, spanning lags and reference dates. So it would require multiple files spanning multiple as-of_datetimes. But this is heavy backend data analysis, so it's okay to require many files.

4. Train forecasting models that use the most recent available estimates as covariates. That's like (1) above, but for all available as-of_datetimes. Acceptable for training, I think.

So (1) is 1 file, and (3) and (4) can afford to take more time. My remaining concern is the map (2). To solve that, we could:
A. store all states/counties in the same file, increasing the size of the file. Do we do that for any of the other signals?
B. store (the 50% quantile only) a second time, grouped by regions.
C. store it as an additional indicator in Epidata. Think of it as another (corrected) variant of the original signal.