use_course()

[jennybc]

Fixes #132 use_course()

Goal:

• Write a shortlink on a whiteboard at a workshop
• Large, diverse group of participants execute usethis::use_course(SHORTLINK)
• A folder's worth of files, from either DropBox or GitHub, is downloaded onto their computer, to a location they are urged to choose and notice!!!!, probably the Desktop, and opened in an OS-appropriate file browser.

---

Usage with a GitHub ZIP behind a bit.ly shortlink. It's a package but you get the idea.

> devtools::load_all("~/rrr/usethis")
Loading usethis

> usethis <- use_course("http://bit.ly/usethis-shortlink-example")
A ZIP file named:
  'r-lib-usethis-v1.1.0-68-g390e05b.zip'
will be downloaded to this folder:
  '~/Desktop'
Prefer a different location? Cancel, try again, and specify `destdir`.

Proceed with this download?
1: I agree
2: Nope
3: Hell no

Selection: 1
✔ Downloading ZIP file to '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'
✔ Unpacking ZIP file into '/Users/jenny/Desktop/r-lib-usethis-390e05b/' (166 files extracted)
Shall we delete the ZIP file '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'?
1: Yeah
2: No way
3: No

Selection: 1
✔ Deleting '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'
✔ Opening '/Users/jenny/Desktop/r-lib-usethis-390e05b/' in the file manager

> ## r-lib-usethis-390e05b is opened in the Finder right here!

> list.files(usethis, all.files = TRUE, recursive = TRUE) %>% sample(10)
 [1] "R/rstudio.R"               "usethis.Rproj"            
 [3] "tests/testthat/helper.R"   "man/use_pipe.Rd"          
 [5] "R/pkgdown.R"               "inst/templates/travis.yml"
 [7] "R/news.R"                  "man/proj_get.Rd"          
 [9] "revdep/.gitignore"         "man/use_revdep.Rd"        

---

Usage with a DropBox ZIP from recent @hadley workshop. Slow because many big Keynote files. Realistic/worst case.

> devtools::load_all("~/rrr/usethis")
Loading usethis

> system.time(
+ hadley <- use_course(
+ "https://www.dropbox.com/sh/ofc1gifr77ofej8/AACuBrToN1Yjo_ZxWfrYnEbJa?dl=1"
+ )
+ )
A ZIP file named:
  '17-tidy-tools.zip'
will be downloaded to this folder:
  '~/Desktop'
Prefer a different location? Cancel, try again, and specify `destdir`.

Proceed with this download?
1: Absolutely
2: No
3: Not yet

Selection: 1
✔ Downloading ZIP file to '~/Desktop/17-tidy-tools.zip'
✔ Unpacking ZIP file into '~/Desktop/17-tidy-tools' (64 files extracted)
Shall we delete the ZIP file '~/Desktop/17-tidy-tools.zip'?
1: Hell no
2: I forget
3: For sure

Selection: 3
✔ Deleting '~/Desktop/17-tidy-tools.zip'
✔ Opening '~/Desktop/17-tidy-tools' in the file manager
   user  system elapsed 
  0.853   0.795  32.533 

> ## 17-tidy-tools is opened in the Finder right here!

> list.files(hadley, all.files = TRUE, recursive = TRUE)
 [1] "0-welcome.key"                                   
 [2] "1-preliminaries.key"                             
 [3] "2-packages.key"                                  
 [4] "3-test.key"                                      
 [5] "4-api-best-practices.key"                        
 [6] "5-fp.key"                                        
 [7] "6-oo.key"                                        
 [8] "7-tidy-eval.key"                                 
 [9] "8-document.key"                                  
[10] "9-share.key"                                     
[11] "abstract.md"                                     
[12] "jenny-notes.Rmd"                                 
[13] "tidy-tools/1-preliminaries.pdf"                  
[14] "tidy-tools/2-packages.pdf"                       
[15] "tidy-tools/3-test.pdf"                           
[16] "tidy-tools/4-api-best-practices.pdf"             
[17] "tidy-tools/5-fp.pdf"                             
[18] "tidy-tools/6-oo.pdf"                             
[19] "tidy-tools/7-tidy-eval.pdf"                      
[20] "tidy-tools/8-document.pdf"                       
[21] "tidy-tools/9-share.pdf"                          
[22] "tidy-tools/hadcol/.gitignore"                    
[23] "tidy-tools/hadcol/.Rbuildignore"                 
[24] "tidy-tools/hadcol/DESCRIPTION"                   
[25] "tidy-tools/hadcol/hadcol.Rproj"                  
[26] "tidy-tools/hadcol/man/add_col.Rd"                
[27] "tidy-tools/hadcol/man/add_cols.Rd"               
[28] "tidy-tools/hadcol/NAMESPACE"                     
[29] "tidy-tools/hadcol/R/add_col.R"                   
[30] "tidy-tools/hadcol/R/add_cols.R"                  
[31] "tidy-tools/hadcol/tests/testthat.R"              
[32] "tidy-tools/hadcol/tests/testthat/test-add-col.R" 
[33] "tidy-tools/hadcol/tests/testthat/test-add-cols.R"
[34] "tidy-tools/hadcol/vignettes/.gitignore"          
[35] "tidy-tools/hadcol/vignettes/addcols.Rmd"         
[36] "tidy-tools/mylittlepackage/.gitignore"           
[37] "tidy-tools/mylittlepackage/.Rbuildignore"        
[38] "tidy-tools/mylittlepackage/DESCRIPTION"          
[39] "tidy-tools/mylittlepackage/mylittlepackage.Rproj"
[40] "tidy-tools/mylittlepackage/NAMESPACE"            
[41] "tidy-tools/mylittlepackage/R/grid_function.R"    
[42] "tidy-tools/mylittlepackage/R/hello.R"            
[43] "tidy-tools/mylittlepackage/R/rpony.R"            
[44] "tidy-tools/safely/.gitignore"                    
[45] "tidy-tools/safely/.Rbuildignore"                 
[46] "tidy-tools/safely/DESCRIPTION"                   
[47] "tidy-tools/safely/NAMESPACE"                     
[48] "tidy-tools/safely/R/safely.R"                    
[49] "tidy-tools/safely/R/utils.R"                     
[50] "tidy-tools/safely/safely.Rproj"                  
[51] "welcome.md"    

[jennybc]

I figure this should be very rigid for now.

[jimhester]

If you are attaching httr, why are you not using it to download the file?

[jimhester]

I think the answer to my question is to auto-find the filename.

I guess it feels like this code should really be in httr, not here, maybe as a extension on httr::write_disk().

If we do keep it here, because we don't plan on a httr release in the near future, I think it would be better to write a write_disk() function and use it in a httr::GET() request.

[jennybc]

Doing this outside of httr, i.e. without access to unexported functions, doesn't look terribly practical. I think I can but it will definitely take me longer. How important is this @jimhester?

[jimhester]

Not that important, but how important is it to have this auto naming in usethis in the first place?

[jennybc]

Very important for this workshop use case!

I know it's hard to believe, but getting a large group of people to download a specific set of files to a folder on their computer that

a) has a sane name
b) they can find again
c) includes ALL the files and no other files

is astonishingly difficult. And I am going to get them to do it, so help me God. Or die trying.

[jennybc]

Too rigid? A filename could potentially also be generated from either the input url or dl$url.

[jimhester]

You can rename non-empty directories, I think the rename error is because the buzzy directory is not empty.

[jimhester]

path_common() I don't think will help here, it finds the common prefix starting at the root, not a common stem anywhere.

[hadley]

You could split the paths into individual directories, and iteratively peel off common prefixes.

[jimhester]

This should be sub() not gsub(), you are only doing at most one substitution per string, and the conditional is not needed, just use the substitution directly.

test <- function(cd) {
  sub("^\"(.+)\"$", "\\1", cd)
}
test("foo/bar")
#> [1] "foo/bar"
test('"foo/bar"')
#> [1] "foo/bar"
test('foo/"bar"')
#> [1] "foo/\"bar\""

Created on 2018-01-09 by the reprex package (v0.1.1.9000).

[jimhester]

This should be sub() it will only match (once) at the start of the string.

[jimhester]

👍

[jennybc]

Note to self: it would be really nice to show progress from download_zip().

[jennybc]

@jimhester Do you agree it's not possible / easy for me to get progress, given I am using curl::curl_fetch_memory()? That's my assessment based on the C code.

[jennybc]

I've made another major advance, so any comments are welcome. However, I'm not twiddling my thumbs. The usage example at the very top is has been updated for current state of things.

[hadley]

Shortlink = bit.ly?

[jennybc]

bit.ly more obvious now

[hadley]

Would it be better to default to the desktop? (Since everyone can find that)

[jennybc]

Good point!

[jennybc]

Done d06440d. Usage examples at very top show this now.

[hadley]

Invisibly?

[hadley]

Needs example with shortlink too

[jennybc]

I was reluctant to commit to this because I want the examples to work. But OK. I'll put a shortlink and make sure it works ... today.

[hadley]

Can we do this automatically?

[jennybc]

Do you mean purely mechanically, i.e. on the front-end?

If yes: I could special-case DropBox and check the input url and make this substitution.

Otherwise, I don't think so. If input is a shortlink, we have no visibility into the redirects, to get at this URL and fix it. The failure mode is also non-specific (Error: Download does not have MIME type 'application/zip').

[jennybc]

I have now learned this:

Information about any short bitly URL http://bit.ly/x is available at http://bit.ly/x+ (that is, the URL with a plus sign appended), for example http://bit.ly/1sNZMwL+. This allows users to see and check the long URL before visiting it.

It's not designed for programmatic use but may be helpful. In any case, probably not doing this now.

[hadley]

Again, might be better to wait?

[hadley]

@jimhester is it worth having something like this in fs? Discarding paths matching globs is fairly common

[jimhester]

We could probably put something in there, we already do filtering in dir_ls(), so it wouldn't be too much work to wrap it in function.

[jimhester]

FWIW just added fs::path_filter() function r-lib/fs@195c8e4

[jennybc]

Also, is my unzip-ignoring of .RData going to be an ugly surprise for anyone? It's the only here that I wondered about.

[jennybc]

At least, I've mentioned in the docs now.

[jimhester]

I think keep() could be rewritten somewhat more nicely.

In the following example keep_old is the current implementation, keep_lgl() is a tweaked form with the same return value. keep_re() simply returns the files to be kept rather than a logical vector and keep() uses fs::path_split() and does not use any regular expression. They all have equivalent output.

keep_old <- function(file,
                     ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {

  ignores <- paste0("(\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z)")
  !any(vapply(ignores, function(x) grepl(x, file, perl = TRUE), logical(1)))
}

keep_lgl <- function(file,
                     ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  ignores <- paste0("((\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z))", collapse = "|")
  !grepl(ignores, file, perl = TRUE)
}

library(testthat)
files <- c("foo", "bar", ".Rproj.user", ".git", "/.git", "/.git/", ".git/",
  "foo/.git", ".git", ".git/config", ".git/objects/06/3d3gysle", ".Rproj.user",
  ".Rproj.user/123jkl/persistent-state", ".Rhistory", ".RData", ".gitignore", "a/.gitignore", "foo.Rproj")
for (f in files) {
  expect_identical(keep_lgl(!!f), keep_old(!!f))
}

keep_re <- function(file,
                    ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  ignores <- paste0("((\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z))", collapse = "|")
  grep(ignores, file, perl = TRUE, value = TRUE, invert = TRUE)
}
keep_re(files)
#> [1] "foo"          "bar"          ".gitignore"   "a/.gitignore"
#> [5] "foo.Rproj"
expect_equal(keep_re(files), files[vapply(files, keep_lgl, logical(1))])

keep <- function(file,
                 ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  file[vapply(fs::path_split(file), function(p) !any(p %in% ignores), logical(1))]
}

keep(files)
#> [1] "foo"          "bar"          ".gitignore"   "a/.gitignore"
#> [5] "foo.Rproj"
expect_equal(keep(files), keep_re(files))

Created on 2018-01-12 by the reprex package (v0.1.1.9000).

[jennybc]

Thanks, adopted in 5275eff.

[jennybc]

I think I've implemented all the feedback. Usage example updated to reflect new default to ~/Desktop/ and usage with a bit.ly shortlink.

[hadley]

I think you should feel free to merge at this point; I'm sure we'll discover problems in the future, but this seems like a substantial amount of useful work.

[jaredlander]

Starting to use this now. First public attempt next week. Can we follow the download with a call to rstudioapi::openProject? That would make it easier for students rather than having to figure out how to open the project themselves. I would simply make it the next line in the script, but the destdir is unknown if they decide to change it.

[jaredlander]

Actually, just realized I can assign projName <- use_course(...) then rstudio::openProject(projName). Might still be a nice convenient functionality for users.

[jennybc]

We made a conscious choice to NOT create and launch an RStudio Project with use_course().

But I could be convinced. And/or we could make another function in the create_*() family that is use_course() + create & launch RStudio Project.

Thoughts, @hadley?

We also learned at rstudio::conf that hitting a GitHub repo with use_course() may be superior to DropBox. We had several people whose locked-down corporate laptops had trouble with DropBox but worked fine with GitHub. That's just a usage tip.

Also, a couple people needed to update the curl and backports packages, so obviously I need to determine and enforce minimum versions of those dependencies.

[jaredlander]

I built a repo for a course I'm teaching. The README has them install some packages, run use_course, open the project and then source a script that came with the repo to download all the data. https://github.com/jaredlander/LiveFebruary2018

[jennybc]

Looks good! I'd love to get feedback on how it works for you. Just so you know, you can use a shortlink with use_course(), if they're going to have to type it and you want to make that less error-prone.

[hadley]

Maybe we should automatically open a .Rproj file if there's one in the root directory?

[jaredlander]

Used it last week and it was HUGE improvement over other ways I've tried to set up a class. Everyone had a project with the right structure so we no longer had path issues.

[jennybc]

Used it last week and it was HUGE improvement over other ways I've tried to set up a class. Everyone had a project with the right structure so we no longer had path issues.

This brings me actual joy 😃, thanks for letting me know! The dev version now opens the folder as an RStudio project if you decided to include an .Rproj file.

[jaredlander]

I built a package to generate a repo skeleton. This way instructors can easily create a new repo with a few lines of code. Then the user can go and do use_course on that generated repo.

This package creates the project on disk, populates the README with the desired packages and builds the file that students source to download data. Then it pushes all of that to GitHub so it is easily accessed by students.

Please see what you think: https://github.com/jaredlander/RepoGenerator