Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use_course() #196

Merged
merged 52 commits into from Jan 13, 2018
Merged

use_course() #196

merged 52 commits into from Jan 13, 2018

Conversation

jennybc
Copy link
Member

@jennybc jennybc commented Jan 8, 2018

Fixes #132 use_course()

Goal:

  • Write a shortlink on a whiteboard at a workshop
  • Large, diverse group of participants execute usethis::use_course(SHORTLINK)
  • A folder's worth of files, from either DropBox or GitHub, is downloaded onto their computer, to a location they are urged to choose and notice!!!!, probably the Desktop, and opened in an OS-appropriate file browser.

Usage with a GitHub ZIP behind a bit.ly shortlink. It's a package but you get the idea.

> devtools::load_all("~/rrr/usethis")
Loading usethis

> usethis <- use_course("http://bit.ly/usethis-shortlink-example")
A ZIP file named:
  'r-lib-usethis-v1.1.0-68-g390e05b.zip'
will be downloaded to this folder:
  '~/Desktop'
Prefer a different location? Cancel, try again, and specify `destdir`.

Proceed with this download?
1: I agree
2: Nope
3: Hell no

Selection: 1Downloading ZIP file to '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'Unpacking ZIP file into '/Users/jenny/Desktop/r-lib-usethis-390e05b/' (166 files extracted)
Shall we delete the ZIP file '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'?
1: Yeah
2: No way
3: No

Selection: 1Deleting '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'Opening '/Users/jenny/Desktop/r-lib-usethis-390e05b/' in the file manager

> ## r-lib-usethis-390e05b is opened in the Finder right here!

> list.files(usethis, all.files = TRUE, recursive = TRUE) %>% sample(10)
 [1] "R/rstudio.R"               "usethis.Rproj"            
 [3] "tests/testthat/helper.R"   "man/use_pipe.Rd"          
 [5] "R/pkgdown.R"               "inst/templates/travis.yml"
 [7] "R/news.R"                  "man/proj_get.Rd"          
 [9] "revdep/.gitignore"         "man/use_revdep.Rd"        

Usage with a DropBox ZIP from recent @hadley workshop. Slow because many big Keynote files. Realistic/worst case.

> devtools::load_all("~/rrr/usethis")
Loading usethis

> system.time(
+ hadley <- use_course(
+ "https://www.dropbox.com/sh/ofc1gifr77ofej8/AACuBrToN1Yjo_ZxWfrYnEbJa?dl=1"
+ )
+ )
A ZIP file named:
  '17-tidy-tools.zip'
will be downloaded to this folder:
  '~/Desktop'
Prefer a different location? Cancel, try again, and specify `destdir`.

Proceed with this download?
1: Absolutely
2: No
3: Not yet

Selection: 1Downloading ZIP file to '~/Desktop/17-tidy-tools.zip'Unpacking ZIP file into '~/Desktop/17-tidy-tools' (64 files extracted)
Shall we delete the ZIP file '~/Desktop/17-tidy-tools.zip'?
1: Hell no
2: I forget
3: For sure

Selection: 3Deleting '~/Desktop/17-tidy-tools.zip'Opening '~/Desktop/17-tidy-tools' in the file manager
   user  system elapsed 
  0.853   0.795  32.533 

> ## 17-tidy-tools is opened in the Finder right here!

> list.files(hadley, all.files = TRUE, recursive = TRUE)
 [1] "0-welcome.key"                                   
 [2] "1-preliminaries.key"                             
 [3] "2-packages.key"                                  
 [4] "3-test.key"                                      
 [5] "4-api-best-practices.key"                        
 [6] "5-fp.key"                                        
 [7] "6-oo.key"                                        
 [8] "7-tidy-eval.key"                                 
 [9] "8-document.key"                                  
[10] "9-share.key"                                     
[11] "abstract.md"                                     
[12] "jenny-notes.Rmd"                                 
[13] "tidy-tools/1-preliminaries.pdf"                  
[14] "tidy-tools/2-packages.pdf"                       
[15] "tidy-tools/3-test.pdf"                           
[16] "tidy-tools/4-api-best-practices.pdf"             
[17] "tidy-tools/5-fp.pdf"                             
[18] "tidy-tools/6-oo.pdf"                             
[19] "tidy-tools/7-tidy-eval.pdf"                      
[20] "tidy-tools/8-document.pdf"                       
[21] "tidy-tools/9-share.pdf"                          
[22] "tidy-tools/hadcol/.gitignore"                    
[23] "tidy-tools/hadcol/.Rbuildignore"                 
[24] "tidy-tools/hadcol/DESCRIPTION"                   
[25] "tidy-tools/hadcol/hadcol.Rproj"                  
[26] "tidy-tools/hadcol/man/add_col.Rd"                
[27] "tidy-tools/hadcol/man/add_cols.Rd"               
[28] "tidy-tools/hadcol/NAMESPACE"                     
[29] "tidy-tools/hadcol/R/add_col.R"                   
[30] "tidy-tools/hadcol/R/add_cols.R"                  
[31] "tidy-tools/hadcol/tests/testthat.R"              
[32] "tidy-tools/hadcol/tests/testthat/test-add-col.R" 
[33] "tidy-tools/hadcol/tests/testthat/test-add-cols.R"
[34] "tidy-tools/hadcol/vignettes/.gitignore"          
[35] "tidy-tools/hadcol/vignettes/addcols.Rmd"         
[36] "tidy-tools/mylittlepackage/.gitignore"           
[37] "tidy-tools/mylittlepackage/.Rbuildignore"        
[38] "tidy-tools/mylittlepackage/DESCRIPTION"          
[39] "tidy-tools/mylittlepackage/mylittlepackage.Rproj"
[40] "tidy-tools/mylittlepackage/NAMESPACE"            
[41] "tidy-tools/mylittlepackage/R/grid_function.R"    
[42] "tidy-tools/mylittlepackage/R/hello.R"            
[43] "tidy-tools/mylittlepackage/R/rpony.R"            
[44] "tidy-tools/safely/.gitignore"                    
[45] "tidy-tools/safely/.Rbuildignore"                 
[46] "tidy-tools/safely/DESCRIPTION"                   
[47] "tidy-tools/safely/NAMESPACE"                     
[48] "tidy-tools/safely/R/safely.R"                    
[49] "tidy-tools/safely/R/utils.R"                     
[50] "tidy-tools/safely/safely.Rproj"                  
[51] "welcome.md"    

@jennybc jennybc requested review from hadley and jimhester Jan 8, 2018
R/course.R Outdated
httr::stop_for_status(dl$status_code)
stopifnot(
grepl("^https://dl.dropboxusercontent.com/content_link_zip/", dl$url) ||
grepl("^https://codeload.github.com", dl$url)
Copy link
Member Author

@jennybc jennybc Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I figure this should be very rigid for now.

Copy link
Member

@jimhester jimhester Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are attaching httr, why are you not using it to download the file?

Copy link
Member

@jimhester jimhester Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the answer to my question is to auto-find the filename.

I guess it feels like this code should really be in httr, not here, maybe as a extension on httr::write_disk().

If we do keep it here, because we don't plan on a httr release in the near future, I think it would be better to write a write_disk() function and use it in a httr::GET() request.

Copy link
Member Author

@jennybc jennybc Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing this outside of httr, i.e. without access to unexported functions, doesn't look terribly practical. I think I can but it will definitely take me longer. How important is this @jimhester?

Copy link
Member

@jimhester jimhester Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that important, but how important is it to have this auto naming in usethis in the first place?

Copy link
Member Author

@jennybc jennybc Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very important for this workshop use case!

I know it's hard to believe, but getting a large group of people to download a specific set of files to a folder on their computer that

a) has a sane name
b) they can find again
c) includes ALL the files and no other files

is astonishingly difficult. And I am going to get them to do it, so help me God. Or die trying.

R/course.R Outdated
hh <- curl::parse_headers_list(dl$headers)
stopifnot(hh[["content-type"]] == "application/zip")
content_disposition <- hh[["content-disposition"]]
stopifnot(!is.null(content_disposition), nzchar(content_disposition))
Copy link
Member Author

@jennybc jennybc Jan 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too rigid? A filename could potentially also be generated from either the input url or dl$url.

@jimhester
Copy link
Member

@jimhester jimhester commented Jan 8, 2018

You can rename non-empty directories, I think the rename error is because the buzzy directory is not empty.

@jimhester
Copy link
Member

@jimhester jimhester commented Jan 8, 2018

path_common() I don't think will help here, it finds the common prefix starting at the root, not a common stem anywhere.

@hadley
Copy link
Member

@hadley hadley commented Jan 8, 2018

You could split the paths into individual directories, and iteratively peel off common prefixes.

R/course.R Outdated
## I know I could use regex and lookahead but this is easier for me to
## maintain
if (grepl("^\"", cd) && grepl("\"$", cd)) {
cd <- gsub("^\"(.+)\"$", "\\1", cd)
Copy link
Member

@jimhester jimhester Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be sub() not gsub(), you are only doing at most one substitution per string, and the conditional is not needed, just use the substitution directly.

test <- function(cd) {
  sub("^\"(.+)\"$", "\\1", cd)
}
test("foo/bar")
#> [1] "foo/bar"
test('"foo/bar"')
#> [1] "foo/bar"
test('foo/"bar"')
#> [1] "foo/\"bar\""

Created on 2018-01-09 by the reprex package (v0.1.1.9000).

R/course.R Outdated
}
message("content-disposition:\n", cd)

cd <- gsub("^attachment;\\s*", "", cd, ignore.case = TRUE)
Copy link
Member

@jimhester jimhester Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be sub() it will only match (once) at the start of the string.

jimhester
Copy link
Member

@jimhester jimhester commented on 09cb6cd Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@jennybc
Copy link
Member Author

@jennybc jennybc commented Jan 9, 2018

Note to self: it would be really nice to show progress from download_zip().

@jennybc
Copy link
Member Author

@jennybc jennybc commented Jan 9, 2018

@jimhester Do you agree it's not possible / easy for me to get progress, given I am using curl::curl_fetch_memory()? That's my assessment based on the C code.

@jennybc
Copy link
Member Author

@jennybc jennybc commented Jan 10, 2018

I've made another major advance, so any comments are welcome. However, I'm not twiddling my thumbs. The usage example at the very top is has been updated for current state of things.

@jennybc
Copy link
Member Author

@jennybc jennybc commented Jan 12, 2018

I consider this ready for review now.

It works on my Windows machine 🎉.

R/course.R Outdated
#' Special-purpose function to download a folder of course materials. The only
#' demand on the user is to confirm or specify where the new folder should be
#' stored. Workflow:
#' * User calls `use_course("SHORTLINK-GOES-HERE")`.
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shortlink = bit.ly?

Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bit.ly more obvious now

R/course.R Outdated
#' @param url Link to a ZIP file containing the materials, possibly behind a
#' shortlink. Function developed with DropBox and GitHub in mind, but should
#' work for ZIP files generally. See [download_zip()] for more.
#' @param destdir The new folder is stored here. Defaults to working directory.
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to default to the desktop? (Since everyone can find that)

Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!

Copy link
Member Author

@jennybc jennybc Jan 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done d06440d. Usage examples at very top show this now.

R/course.R Outdated
#' work for ZIP files generally. See [download_zip()] for more.
#' @param destdir The new folder is stored here. Defaults to working directory.
#'
#' @return Path to the new directory holding the course materials.
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Invisibly?

R/course.R Outdated
#' ## demo with a small CRAN package available in various places
#'
#' ## from CRAN
#' use_course("https://cran.r-project.org/bin/windows/contrib/3.4/rematch2_2.0.1.zip")
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs example with shortlink too

Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was reluctant to commit to this because I want the examples to work. But OK. I'll put a shortlink and make sure it works ... today.

#' ```
#' https://www.dropbox.com/sh/12345abcde/6789wxyz?dl=0
#' ```
#' Replace the `dl=0` at the end with `dl=1` to create a download link. The ZIP
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this automatically?

Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean purely mechanically, i.e. on the front-end?

If yes: I could special-case DropBox and check the input url and make this substitution.

Otherwise, I don't think so. If input is a shortlink, we have no visibility into the redirects, to get at this URL and fix it. The failure mode is also non-specific (Error: Download does not have MIME type 'application/zip').

Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have now learned this:

Information about any short bitly URL http://bit.ly/x is available at http://bit.ly/x+ (that is, the URL with a plus sign appended), for example http://bit.ly/1sNZMwL+. This allows users to see and check the long URL before visiting it.

It's not designed for programmatic use but may be helpful. In any case, probably not doing this now.

R/course.R Outdated
#' @return Path to the directory holding the unpacked files.
#' @keywords internal
#' @family download functions
#' @export
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, might be better to wait?

R/course.R Outdated
keep <- function(file,
ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
ignores <- paste0("(\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z)")
!any(vapply(ignores, function(x) grepl(x, file, perl = TRUE), logical(1)))
Copy link
Member

@hadley hadley Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimhester is it worth having something like this in fs? Discarding paths matching globs is fairly common

Copy link
Member

@jimhester jimhester Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could probably put something in there, we already do filtering in dir_ls(), so it wouldn't be too much work to wrap it in function.

Copy link
Member

@jimhester jimhester Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW just added fs::path_filter() function r-lib/fs@195c8e4

R/course.R Outdated
}

keep <- function(file,
ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, is my unzip-ignoring of .RData going to be an ugly surprise for anyone? It's the only here that I wondered about.

Copy link
Member Author

@jennybc jennybc Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least, I've mentioned in the docs now.

R/course.R Outdated
invisible(target)
}

keep <- function(file,
Copy link
Member

@jimhester jimhester Jan 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think keep() could be rewritten somewhat more nicely.

In the following example keep_old is the current implementation, keep_lgl() is a tweaked form with the same return value. keep_re() simply returns the files to be kept rather than a logical vector and keep() uses fs::path_split() and does not use any regular expression. They all have equivalent output.

keep_old <- function(file,
                     ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {

  ignores <- paste0("(\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z)")
  !any(vapply(ignores, function(x) grepl(x, file, perl = TRUE), logical(1)))
}

keep_lgl <- function(file,
                     ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  ignores <- paste0("((\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z))", collapse = "|")
  !grepl(ignores, file, perl = TRUE)
}

library(testthat)
files <- c("foo", "bar", ".Rproj.user", ".git", "/.git", "/.git/", ".git/",
  "foo/.git", ".git", ".git/config", ".git/objects/06/3d3gysle", ".Rproj.user",
  ".Rproj.user/123jkl/persistent-state", ".Rhistory", ".RData", ".gitignore", "a/.gitignore", "foo.Rproj")
for (f in files) {
  expect_identical(keep_lgl(!!f), keep_old(!!f))
}

keep_re <- function(file,
                    ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  ignores <- paste0("((\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z))", collapse = "|")
  grep(ignores, file, perl = TRUE, value = TRUE, invert = TRUE)
}
keep_re(files)
#> [1] "foo"          "bar"          ".gitignore"   "a/.gitignore"
#> [5] "foo.Rproj"
expect_equal(keep_re(files), files[vapply(files, keep_lgl, logical(1))])

keep <- function(file,
                 ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  file[vapply(fs::path_split(file), function(p) !any(p %in% ignores), logical(1))]
}

keep(files)
#> [1] "foo"          "bar"          ".gitignore"   "a/.gitignore"
#> [5] "foo.Rproj"
expect_equal(keep(files), keep_re(files))

Created on 2018-01-12 by the reprex package (v0.1.1.9000).

Copy link
Member Author

@jennybc jennybc Jan 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, adopted in 5275eff.

@jennybc
Copy link
Member Author

@jennybc jennybc commented Jan 13, 2018

I think I've implemented all the feedback. Usage example updated to reflect new default to ~/Desktop/ and usage with a bit.ly shortlink.

@hadley
Copy link
Member

@hadley hadley commented Jan 13, 2018

I think you should feel free to merge at this point; I'm sure we'll discover problems in the future, but this seems like a substantial amount of useful work.

@jennybc jennybc merged commit a62c56e into master Jan 13, 2018
5 of 6 checks passed
@jennybc jennybc deleted the use-course branch Jan 14, 2018
@jaredlander
Copy link

@jaredlander jaredlander commented Feb 9, 2018

Starting to use this now. First public attempt next week. Can we follow the download with a call to rstudioapi::openProject? That would make it easier for students rather than having to figure out how to open the project themselves. I would simply make it the next line in the script, but the destdir is unknown if they decide to change it.

@jaredlander
Copy link

@jaredlander jaredlander commented Feb 9, 2018

Actually, just realized I can assign projName <- use_course(...) then rstudio::openProject(projName). Might still be a nice convenient functionality for users.

@jennybc
Copy link
Member Author

@jennybc jennybc commented Feb 9, 2018

We made a conscious choice to NOT create and launch an RStudio Project with use_course().

But I could be convinced. And/or we could make another function in the create_*() family that is use_course() + create & launch RStudio Project.

Thoughts, @hadley?

We also learned at rstudio::conf that hitting a GitHub repo with use_course() may be superior to DropBox. We had several people whose locked-down corporate laptops had trouble with DropBox but worked fine with GitHub. That's just a usage tip.

Also, a couple people needed to update the curl and backports packages, so obviously I need to determine and enforce minimum versions of those dependencies.

@jaredlander
Copy link

@jaredlander jaredlander commented Feb 10, 2018

I built a repo for a course I'm teaching. The README has them install some packages, run use_course, open the project and then source a script that came with the repo to download all the data. https://github.com/jaredlander/LiveFebruary2018

@jennybc
Copy link
Member Author

@jennybc jennybc commented Feb 10, 2018

Looks good! I'd love to get feedback on how it works for you. Just so you know, you can use a shortlink with use_course(), if they're going to have to type it and you want to make that less error-prone.

@hadley
Copy link
Member

@hadley hadley commented Feb 10, 2018

Maybe we should automatically open a .Rproj file if there's one in the root directory?

@jaredlander
Copy link

@jaredlander jaredlander commented Feb 21, 2018

Used it last week and it was HUGE improvement over other ways I've tried to set up a class. Everyone had a project with the right structure so we no longer had path issues.

@jennybc
Copy link
Member Author

@jennybc jennybc commented Feb 21, 2018

Used it last week and it was HUGE improvement over other ways I've tried to set up a class. Everyone had a project with the right structure so we no longer had path issues.

This brings me actual joy 😃, thanks for letting me know! The dev version now opens the folder as an RStudio project if you decided to include an .Rproj file.

@jaredlander
Copy link

@jaredlander jaredlander commented May 7, 2018

I built a package to generate a repo skeleton. This way instructors can easily create a new repo with a few lines of code. Then the user can go and do use_course on that generated repo.

This package creates the project on disk, populates the README with the desired packages and builds the file that students source to download data. Then it pushes all of that to GitHub so it is easily accessed by students.

Please see what you think: https://github.com/jaredlander/RepoGenerator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants