`epi_archive$other_keys` is empty

[dsweber2]

Apparently not intended behavior. The underlying DT has the right keys, it's just not recorded in metadata at all

Working example

dv <- pub_covidcast(
  source = "doctor-visits",
  signals = "smoothed_adj_cli",
  geo_type = "state",
  time_type = "day",
  geo_values = "ca,fl,ny,tx",
  time_values = epirange(20200601, 20211201),
  issues = epirange(20200601, 20211201)
)
archive <- dv %>%
  select(geo_value, time_value, version = issue, lag, percent_cli = value) %>%
  as_epi_archive(compactify = TRUE, other_keys = c("lag"))

> archive$DT
Key: <geo_value, time_value, lag, version>
        geo_value time_value    version   lag percent_cli
           <char>     <Date>     <Date> <num>       <num>
     1:        ca 2020-06-01 2020-06-06     5    2.140116
     2:        ca 2020-06-01 2020-06-07     6    2.140116
     3:        ca 2020-06-01 2020-06-08     7    2.140379
     4:        ca 2020-06-01 2020-06-09     8    2.114430
     5:        ca 2020-06-01 2020-06-10     9    2.133677
    ---                                                  
117144:        tx 2021-11-23 2021-11-29     6    2.159704
117145:        tx 2021-11-24 2021-11-27     3    2.024028
117146:        tx 2021-11-24 2021-11-29     5    1.937911
117147:        tx 2021-11-25 2021-11-29     4    1.866631
117148:        tx 2021-11-26 2021-11-29     3    1.858596

> names(archive)
[1] "DT"                         "geo_type"                  
[3] "time_type"                  "additional_metadata"       
[5] "clobberable_versions_start" "versions_end"              

[brookslogan]

The other_keys are reflected in the DT's key. Does seem a bit unnatural to access. The solution may be an other_keys() or get_other_keys() method that works on both epi_dfs and epi_archives. I'd also like an epi_keys() / get_epi_keys() method, but that already exists in epipredict with a different meaning than I would have assigned (time_value is included for epi_dfs, and maybe there's not an archive impl yet), and not sure what other name to use for what I'd have called epi_keys (= geo_value, !!!other_keys for both epi_dfs and epi_archives).

Is there anywhere internally where we try to pull $other_keys / [["other_keys"]]? I don't see any right now in (a recent version of) epiprocess.

[dsweber2]

The solution may be an other_keys() or get_other_keys() method that works on both epi_dfs and epi_archives

remembering attributes(x)$metadata always adds friction for me, so I'd definitely like this feature.

what other name to use for what I'd have called epi_keys

maybe non_time_keys? fwiw, I would have assumed epi_keys meant the entire index, and not just some subset of it.

Is there anywhere internally where we try to pull $other_keys / [["other_keys"]]? I don't see any right now in (a recent version of) epiprocess.

I also can't find any examples for epi_archive. There's a couple for epi_df:

./R/key_colnames.R:27:  other_keys <- attr(x, "metadata")$other_keys
./R/key_colnames.R:33:  other_keys <- attr(x, "metadata")$other_keys
./R/methods-epi_df.R:31:  if (missing(key)) key <- c("geo_value", attributes(x)$metadata$other_keys)
./R/methods-epi_df.R:165:  old_other_keys <- attr(template, "metadata")$other_keys
./R/methods-epi_df.R:166:  attr(res, "metadata")$other_keys <- old_other_keys[old_other_keys %in% cn]
./R/epi_df.R:307:    additional_metadata$other_keys <- unique(
./R/epi_df.R:308:      c(additional_metadata$other_keys, tsibble_other_keys)

[brookslogan]

Is this addressed by the recent key-related functions added? (I still kind of wish for an epikey (= geo x demo) tidyselector, but that looks to be hard to name correctly.)

[dsweber2]

do you mean key_colnames? I haven't looked at this in a while

[brookslogan]

Yes.. but it's not what I expected...

epiprocess::key_colnames(archive_cases_dv_subset)
[1] "geo_value"  "time_value"

this is neither the unique key of the archive, nor what I call the epikey.

Also, the implementation is buggy if we have other_keys anyway

epiprocess:::key_colnames.epi_archive
#> function(x, exclude = character(0L), ...) {
#>   assert_character(exclude)
#>   other_keys <- attr(x, "metadata")$other_keys
#>   setdiff(c("geo_value", other_keys, "time_value"), exclude)
#> }

but we don't store epi_archive's other_keys inside a metadata attribute, but instead directly as list elements [redundantly, since it's already encoded in the DT key as noted]. Actually, seems like the title and example of this PR has been addressed

archive_cases_dv_subset$other_keys
#> character(0)

library(epidatr)
#> ! epidatr cache is being used (set env var EPIDATR_USE_CACHE=FALSE if not intended).
#> ℹ The cache directory is ~/.cache/R/epidatr.
#> ℹ The cache will be cleared after 14 days and will be pruned if it exceeds 4096 MB.
#> ℹ The log of cache transactions is stored at ~/.cache/R/epidatr/logfile.txt.
dv <- pub_covidcast(
  source = "doctor-visits",
  signals = "smoothed_adj_cli",
  geo_type = "state",
  time_type = "day",
  geo_values = "ca,fl,ny,tx",
  time_values = epirange(20200601, 20211201),
  issues = epirange(20200601, 20211201)
)
archive <- dv %>%
  select(geo_value, time_value, version = issue, lag, percent_cli = value) %>%
  as_epi_archive(compactify = TRUE, other_keys = c("lag"))
archive$other_keys
#> [1] "lag"

Going to return to this later.

• todo: close this & file something else for above?

[dsweber2]

yeah it looks like this has actually been addressed; whether key_colnames(archive_cases_dv_subset) returns the right thing for archives is a different issue.

[brookslogan]

Side note: I assume the sample code was just for simple illustration, but in real code we probably wouldn't want to use lag in other_keys, since it would disable compactification and mess up the daily -> weekly archive transformation.