Issues installing evalcast with remotes::install_github

[keerstorm]

Hey there. Came to this from the PNAS article and I'm trying to run the code but can't get started.

Following the recommended install yielded

> remotes::install_github("cmu-delphi/covidcast@evalcast-0.3", subdir = "R-packages/evalcast")
Downloading GitHub repo cmu-delphi/covidcast@evalcast-0.3
Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
  download from 'https://api.github.com/repos/cmu-delphi/covidcast/tarball/evalcast-0.3' failed

Tried a few variations found with no luck.

> remotes::install_github("cmu-delphi/covidcast", subdir = "R-packages/evalcast")
Downloading GitHub repo cmu-delphi/covidcast@HEAD
Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
  download from 'https://api.github.com/repos/cmu-delphi/covidcast/tarball/HEAD' failed
> remotes::install_github("cmu-delphi/covidcast", ref="evalcast-0.3", subdir = "R-packages/evalcast")
Downloading GitHub repo cmu-delphi/covidcast@evalcast-0.3
Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
  download from 'https://api.github.com/repos/cmu-delphi/covidcast/tarball/evalcast-0.3' failed
> remotes::install_github("cmu-delphi/covidcast", ref = "main", subdir = "R-packages/evalcast")
Downloading GitHub repo cmu-delphi/covidcast@main
Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
  download from 'https://api.github.com/repos/cmu-delphi/covidcast/tarball/main' failed

Would love help. Thanks!

[capnrefsmmat]

If you open those URLs (such as https://api.github.com/repos/cmu-delphi/covidcast/tarball/HEAD) in your browser, do they download a large file successfully?

I suspect this is the same problem as #522, where the repository is so big that download.file automatically times out the request before it finishes, even though it's proceeding just fine. You may be able to clone this repository manually and install that way, if you're familiar with Git.

[keerstorm]

Thanks for the swift response. I would have expected this to work after downloading from the link you supplied, but it's erroring out as well

> install.packages('cmu-delphi-covidcast-evalcast-0.3-20-g51754be.tar.gz', lib=NULL, repos = NULL)
Warning in untar2(tarfile, files, list, exdir, restore_times) :
  skipping pax global extended headers
ERROR: cannot extract package from ‘cmu-delphi-covidcast-evalcast-0.3-20-g51754be.tar.gz’
Warning in install.packages :
  installation of package ‘cmu-delphi-covidcast-evalcast-0.3-20-g51754be.tar.gz’ had non-zero exit status

Cards on the table, I'm a python dev and haven't played around in R for a few years now so apologies for anything obvious I may be missing.

Is there a git command you'd expect to work?

[capnrefsmmat]

The file you downloaded contains the whole repository, including multiple different packages, so install.packages probably doesn't know what to do with it.

If you decompress the file yourself, you should get a covidcast/ directory containing R-packages/evalcast/, and could use install.packages('path/to/covidcast/R-packages/evalcast/', type = "source", repos = NULL), or something like that.

[keerstorm]

Great--thanks!

So, I'm not sure if you're working directly with the team authoring the PNAS article directly, but if you are ....

(and because I can't bug report to the Zenodo page)

... With respect to the forecast/code/README.md , I think it may be worth
adding
remotes::install_github("reichlab/covidHubUtils"),
updating the quantgen install to
remotes::install_github("ryantibs/quantgen", subdir="quantgen"),
and adding zoltyr and arrow to required packages.
Also, I believe qr_lags was intended to be defined in pkgs_and_common.R

Sorry if you're not working on it. Thanks again for the help!

[capnrefsmmat]

Let me just tag @dajmcdon here, since hew was involved in most of the evalcast and forecast work

[dwill023]

I had the same issue as above downloading the evalcast package as stated in https://cmu-delphi.github.io/covidcast/evalcastR/.

So wanted to summarize the steps:

1. install the dependencies:

install.packages("zoltar")
install.packages("arrow")
install.packages("covidcast")
devtools::install_github("gfkse/bettermc")
remotes::install_github("reichlab/covidHubUtils")
remotes::install_github("ryantibs/quantgen", subdir="quantgen")

2. Download the whole package repo: https://api.github.com/repos/cmu-delphi/covidcast/tarball/HEAD
3. Unzip the .tar.gz file:

$ tar -xf cmu-delphi-covidcast-evalcast-0.3-263-g4431561.tar.gz

4. Install the evalcast package by navigating to the R-packages/evalcast directory.

install.package("C:/Downloads/cmu-delphi-covidcast-4431561/R-packages/evalcast")

These steps worked for me.

[dajmcdon]

@capnrefsmmat @ryantibs Is it worth pushing an update to Zenodo re ☝️ ?

• The README has the wrong quantgen installation instructions. Should be remotes::install_github("ryantibs/quantgen", subdir="quantgen") as stated above.
• evalcast is hard to install because it's buried in this repo. (Though also not really supported anymore).
• The string of package dependencies is a bit cumbersome, and it's not clear to me how long the workflow will continue to function unless we pin lots of packages to a specific version.

[dshemetov]

I've been taking care of this repo lately, since I think it's worth shepherding it to a state where it's (relatively) stable and finalized. So I can help come up with installation updates.

I am curious about reproducing the installation difficulties here, since, at least locally, these packages installed smoothly with either devtools::install_github or pak::install.

[dshemetov]

Here's an attempt to repro:

r$> remotes::install_github("cmu-delphi/covidcast", ref = "main",
                             subdir = "R-packages/evalcast")
Using github PAT from envvar GITHUB_PAT
Downloading GitHub repo cmu-delphi/covidcast@main
Error: Failed to install 'covidcast' from GitHub:
  Unknown remote type: bettermc=github
  object 'bettermc=github_remote' of mode 'function' was not found

r$> devtools::install_github("cmu-delphi/covidcast", ref = "main",
                             subdir = "R-packages/evalcast")
Using github PAT from envvar GITHUB_PAT
Downloading GitHub repo cmu-delphi/covidcast@main
Error: Failed to install 'covidcast' from GitHub:
  Unknown remote type: bettermc=github
  object 'bettermc=github_remote' of mode 'function' was not found

r$> pak::pkg_install("cmu-delphi/covidcast/R-packages/evalcast@main")
✔ Updated metadata database: 2.75 MB in 6 files.                                          
✔ Updating metadata database ... done                                      
                                                                           
→ Will install 100 packages.
→ Will update 3 packages.
→ Will download 8 CRAN packages (11.08 MB), cached: 92 (57.06 MB).
→ Will download 3 packages with unknown size.
+ arrow                  12.0.0     [bld][cmp]
+ askpass                1.1        [bld][cmp]
+ assertthat             0.2.1      [bld]
...

It seems that the name=remote syntax in DEPENDENCIES is unfortunately pak-only and we're using it on the main branch here.

[dajmcdon]

@dshemetov I think the problem is due to the size of the repo. See #522.

[dshemetov]

Oh hm. So it looks like devtools::install_github has a download timeout, I wonder if pak does too.

[dwill023]

I got the same error as you have above.

> devtools::install_github("cmu-delphi/covidcast", ref = "main",
+                          subdir = "R-packages/evalcast")
Downloading GitHub repo cmu-delphi/covidcast@main
Error: Failed to install 'evalcast' from GitHub:
  Unknown remote type: bettermc=github
  object 'bettermc=github_remote' of mode 'function' was not found

But when I try with pak::pkg_install("cmu-delphi/covidcast/R-packages/evalcast@main") it works.

r$> pak::pkg_install("cmu-delphi/covidcast/R-packages/evalcast@main")
✔ Updated metadata database: 4.99 MB in 12 files.                                         
✔ Updating metadata database ... done                                      
                                                                             
→ Will update 3 packages.
→ Will download 3 packages with unknown size.
+ bettermc 1.2.2 → 1.1.2 [bld][cmp][dl] (GitHub: ccb0ad5)
+ evalcast 0.3.1 → 0.3.1 [bld][cmp][dl] (GitHub: 4431561)
+ zoltr    0.5.1 → 1.0.0 [bld][cmp][dl] (GitHub: 4e77910)
? Do you want to continue (Y/n) Y
ℹ Getting 3 pkgs with unknown sizes
✔ Got bettermc 1.1.2 (source) (116.22 kB)                                       
✔ Got zoltr 1.0.0 (source) (292.96 kB)                                         
✔ Got evalcast 0.3.1 (source) (182.99 MB)                            
✔ Downloaded 3 packages (183.40 MB)in 31.3s                 
ℹ Packaging zoltr 1.0.0
ℹ Packaging bettermc 1.1.2
✔ Packaged bettermc 1.1.2 (3s)                                                 
ℹ Building bettermc 1.1.2                                                      
✔ Packaged zoltr 1.0.0 (3.5s)                                                  
ℹ Building zoltr 1.0.0                                                         
✔ Built zoltr 1.0.0 (4.9s)                                                     
✔ Installed zoltr 1.0.0 (github::reichlab/zoltr@4e77910) (184ms)                          
✔ Built bettermc 1.1.2 (12.8s)                                                          
✔ Installed bettermc 1.1.2 (github::gfkse/bettermc@ccb0ad5) (179ms)       
ℹ Packaging evalcast 0.3.1                                                
✔ Packaged evalcast 0.3.1 (4.3s)                                        
ℹ Building evalcast 0.3.1                                               
✔ Built evalcast 0.3.1 (7.1s)                                           
✔ Installed evalcast 0.3.1 (github::cmu-delphi/covidcast@4431561) (155ms) 
✔ 1 pkg + 104 deps: kept 98, upd 3, dld 3 (NA B) [1m 30.9s]   

r$> sessionInfo()
R version 4.3.0 (2023-04-21 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    

time zone: America/Denver
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.3.0  tools_4.3.0     rstudioapi_0.14 pak_0.5.1           

[dshemetov]

Thank you for testing that out @dwill023!

UPDATE: I have removed code that made it so only pak is able to install. remotes and devtools should now work as well, so likely no changes needed here.