Add `epi_archive` class

[brookslogan]

Successor of issue #8. Work in progress on PR #22.

Add a separate class to manage version history data for epi_tibbles, called epi_archive or epi_tibble_archive. Make it work as outlined in the issues vignette. Complete the issues vignette to show this functionality.

[brookslogan]

See also the table-diff-objects branch for work toward potentially useful/explanatory diff-of-data-table objects. These may help compact the update.DT objects to only use a minimal number of rows, to validate user input, to update archives with additional data, to provide information for those modeling revisions, etc.

E: (this file contains the current work on diffs)

[brookslogan]

Additional tasks beyond those implied above:

• the delphi-epidata issue data example may not match current epidata as-of query results when values are removed, although some PRs being integrated in covidcast will change them to agree. This discrepancy should be noted.
• the delphi-epidata issue data example might be useful as its own set of functions for this package
• some functions to generate fake archive data for testing
• testthat tests
• geo_type and time_type handling in epi_tibble_archive rather than relying on epi_tibble to later infer the appropriate choices
• decide on future of naming metainfo in the archive; probably issue, geo_value, time_value should all be fixed beforehand
• consider adding data license metadata/data
• consider specializing epi_slide to just use rolling joins to get data in the desired time_value window rather than filtering a full snapshot
• consider changing the way that epi_slide parallels epi_tibble: rather than try to match the output format, implement group_by.epi_archive, make epi_slide go from epi_archives to epi_archives, and have a function to get from an epi_archive to an epi_tibble (as.epi_tibble.epi_archive) for these one-unique-time-per-issue epi_archives

[brookslogan]

Old examples of epi_archive were removed at some point, but the second one was personally useful. I'm just updating it a bit here to give the expected TRUE results on my current system (maybe different all.equal behavior?) and with the latest code.

library(dplyr)
## (delphi.epidata package is on GitHub in cmu-delphi/delphi-epidata-r)

update.df.2 =
  delphi.epidata::covidcast("jhu-csse", "confirmed_incidence_num",
                            "day", "state",
                            delphi.epidata::epirange(12340101,34560101), c("ak","al"),
                            issues = delphi.epidata::epirange(12340101,34560101)) %>%
  delphi.epidata::fetch_tbl()
snapshot.df.2a =
  delphi.epidata::covidcast("jhu-csse", "confirmed_incidence_num",
                            "day", "state",
                            delphi.epidata::epirange(12340101,34560101), c("ak","al"),
                            as_of = 20201014) %>%
  delphi.epidata::fetch_tbl()
snapshot.df.2b =
  delphi.epidata::covidcast("jhu-csse", "confirmed_incidence_num",
                            "day", "state",
                            delphi.epidata::epirange(12340101,34560101), c("ak","al"),
                            as_of = 20201028) %>%
  delphi.epidata::fetch_tbl()

epi.tibble.archive.2 = epiprocess::epi_archive$new(update.df.2)
all_equal(
  as_tibble(epi.tibble.archive.2$epi_df_as_of(as.Date("2020-10-14"))),
  as_tibble(snapshot.df.2a %>>% select(-issue)))
all_equal(
  as_tibble(epi.tibble.archive.2$epi_df_as_of(as.Date("2020-10-28"))),
  as_tibble(snapshot.df.2b %>>% select(-issue)))

Warnings are from #36 or something similar.

[brookslogan]

For preparation of "bleeding-edge" data frames with data as of t+k for every time t, here are a few approaches. Looks like using the existing update.DT (+ an update.lag calculation in case there is no lag column in the input data) is comparable in speed to a more direct approach that would have required a different DT key. [This approaches would need to be updated to use the actual key rather than hardcoded c("geo_value,"time"value","issue"). Plus there will need to be an additional option to add an as_of, to get the bleeding-edge training data as of some version, or a function to call beforehand or after to do the appropriate issue <= as_of filter operation. Plus, this code assumes that issue is a date; that should be verified. Long-term there should probably be an S3 function to take the place of as.integer(issue - time_value) to allow users to work with non-Date issue "numbers".]

update.df =
  delphi.epidata::covidcast("jhu-csse", "confirmed_incidence_num",
                            "day", "state",
                            delphi.epidata::epirange(12340101,34560101), "*",
                            issues = delphi.epidata::epirange(12340101,34560101)) %>%
  delphi.epidata::fetch_tbl()

lag.DT = as.data.table(update.df, key=c("geo_value", "time_value", "lag"))
k = 5

as.of.lag.k.a = unique(lag.DT[lag >= k], by=c("geo_value","time_value"), fromLast=FALSE)

update.DT = as.data.table(update.df, key=c("geo_value", "time_value", "issue"))
set(update.DT, , "update.lag", as.integer(update.DT[["issue"]] - update.DT[["time_value"]]))

as.of.lag.k.b = unique(update.DT[update.lag >= k], by=c("geo_value","time_value"), fromLast=FALSE)
set(as.of.lag.k.b, , "update.lag", NULL)

all_equal(as_tibble(as.of.lag.k.a), as_tibble(as.of.lag.k.b))

microbenchmark::microbenchmark(
                  unique(lag.DT[lag >= k], by=c("geo_value","time_value"), fromLast=FALSE),
                  unique(update.DT[update.lag >= k], by=c("geo_value","time_value"), fromLast=FALSE)[, update.lag := NULL][],
                  times=100L
                )

[ryantibs]

Finally closed! Via #45