Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use_course() #196

Merged
merged 52 commits into from Jan 13, 2018

Conversation

Projects
None yet
4 participants
@jennybc
Copy link
Member

jennybc commented Jan 8, 2018

Fixes #132 use_course()

Goal:

  • Write a shortlink on a whiteboard at a workshop
  • Large, diverse group of participants execute usethis::use_course(SHORTLINK)
  • A folder's worth of files, from either DropBox or GitHub, is downloaded onto their computer, to a location they are urged to choose and notice!!!!, probably the Desktop, and opened in an OS-appropriate file browser.

Usage with a GitHub ZIP behind a bit.ly shortlink. It's a package but you get the idea.

> devtools::load_all("~/rrr/usethis")
Loading usethis

> usethis <- use_course("http://bit.ly/usethis-shortlink-example")
A ZIP file named:
  'r-lib-usethis-v1.1.0-68-g390e05b.zip'
will be downloaded to this folder:
  '~/Desktop'
Prefer a different location? Cancel, try again, and specify `destdir`.

Proceed with this download?
1: I agree
2: Nope
3: Hell no

Selection: 1Downloading ZIP file to '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'Unpacking ZIP file into '/Users/jenny/Desktop/r-lib-usethis-390e05b/' (166 files extracted)
Shall we delete the ZIP file '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'?
1: Yeah
2: No way
3: No

Selection: 1Deleting '~/Desktop/r-lib-usethis-v1.1.0-68-g390e05b.zip'Opening '/Users/jenny/Desktop/r-lib-usethis-390e05b/' in the file manager

> ## r-lib-usethis-390e05b is opened in the Finder right here!

> list.files(usethis, all.files = TRUE, recursive = TRUE) %>% sample(10)
 [1] "R/rstudio.R"               "usethis.Rproj"            
 [3] "tests/testthat/helper.R"   "man/use_pipe.Rd"          
 [5] "R/pkgdown.R"               "inst/templates/travis.yml"
 [7] "R/news.R"                  "man/proj_get.Rd"          
 [9] "revdep/.gitignore"         "man/use_revdep.Rd"        

Usage with a DropBox ZIP from recent @hadley workshop. Slow because many big Keynote files. Realistic/worst case.

> devtools::load_all("~/rrr/usethis")
Loading usethis

> system.time(
+ hadley <- use_course(
+ "https://www.dropbox.com/sh/ofc1gifr77ofej8/AACuBrToN1Yjo_ZxWfrYnEbJa?dl=1"
+ )
+ )
A ZIP file named:
  '17-tidy-tools.zip'
will be downloaded to this folder:
  '~/Desktop'
Prefer a different location? Cancel, try again, and specify `destdir`.

Proceed with this download?
1: Absolutely
2: No
3: Not yet

Selection: 1Downloading ZIP file to '~/Desktop/17-tidy-tools.zip'Unpacking ZIP file into '~/Desktop/17-tidy-tools' (64 files extracted)
Shall we delete the ZIP file '~/Desktop/17-tidy-tools.zip'?
1: Hell no
2: I forget
3: For sure

Selection: 3Deleting '~/Desktop/17-tidy-tools.zip'Opening '~/Desktop/17-tidy-tools' in the file manager
   user  system elapsed 
  0.853   0.795  32.533 

> ## 17-tidy-tools is opened in the Finder right here!

> list.files(hadley, all.files = TRUE, recursive = TRUE)
 [1] "0-welcome.key"                                   
 [2] "1-preliminaries.key"                             
 [3] "2-packages.key"                                  
 [4] "3-test.key"                                      
 [5] "4-api-best-practices.key"                        
 [6] "5-fp.key"                                        
 [7] "6-oo.key"                                        
 [8] "7-tidy-eval.key"                                 
 [9] "8-document.key"                                  
[10] "9-share.key"                                     
[11] "abstract.md"                                     
[12] "jenny-notes.Rmd"                                 
[13] "tidy-tools/1-preliminaries.pdf"                  
[14] "tidy-tools/2-packages.pdf"                       
[15] "tidy-tools/3-test.pdf"                           
[16] "tidy-tools/4-api-best-practices.pdf"             
[17] "tidy-tools/5-fp.pdf"                             
[18] "tidy-tools/6-oo.pdf"                             
[19] "tidy-tools/7-tidy-eval.pdf"                      
[20] "tidy-tools/8-document.pdf"                       
[21] "tidy-tools/9-share.pdf"                          
[22] "tidy-tools/hadcol/.gitignore"                    
[23] "tidy-tools/hadcol/.Rbuildignore"                 
[24] "tidy-tools/hadcol/DESCRIPTION"                   
[25] "tidy-tools/hadcol/hadcol.Rproj"                  
[26] "tidy-tools/hadcol/man/add_col.Rd"                
[27] "tidy-tools/hadcol/man/add_cols.Rd"               
[28] "tidy-tools/hadcol/NAMESPACE"                     
[29] "tidy-tools/hadcol/R/add_col.R"                   
[30] "tidy-tools/hadcol/R/add_cols.R"                  
[31] "tidy-tools/hadcol/tests/testthat.R"              
[32] "tidy-tools/hadcol/tests/testthat/test-add-col.R" 
[33] "tidy-tools/hadcol/tests/testthat/test-add-cols.R"
[34] "tidy-tools/hadcol/vignettes/.gitignore"          
[35] "tidy-tools/hadcol/vignettes/addcols.Rmd"         
[36] "tidy-tools/mylittlepackage/.gitignore"           
[37] "tidy-tools/mylittlepackage/.Rbuildignore"        
[38] "tidy-tools/mylittlepackage/DESCRIPTION"          
[39] "tidy-tools/mylittlepackage/mylittlepackage.Rproj"
[40] "tidy-tools/mylittlepackage/NAMESPACE"            
[41] "tidy-tools/mylittlepackage/R/grid_function.R"    
[42] "tidy-tools/mylittlepackage/R/hello.R"            
[43] "tidy-tools/mylittlepackage/R/rpony.R"            
[44] "tidy-tools/safely/.gitignore"                    
[45] "tidy-tools/safely/.Rbuildignore"                 
[46] "tidy-tools/safely/DESCRIPTION"                   
[47] "tidy-tools/safely/NAMESPACE"                     
[48] "tidy-tools/safely/R/safely.R"                    
[49] "tidy-tools/safely/R/utils.R"                     
[50] "tidy-tools/safely/safely.Rproj"                  
[51] "welcome.md"    

jennybc added some commits Jan 8, 2018

Import curl and httr
These were already indirect dependencies anyway.

@jennybc jennybc requested review from hadley and jimhester Jan 8, 2018

R/course.R Outdated
httr::stop_for_status(dl$status_code)
stopifnot(
grepl("^https://dl.dropboxusercontent.com/content_link_zip/", dl$url) ||
grepl("^https://codeload.github.com", dl$url)

This comment has been minimized.

@jennybc

jennybc Jan 8, 2018

Author Member

I figure this should be very rigid for now.

This comment has been minimized.

@jimhester

jimhester Jan 8, 2018

Member

If you are attaching httr, why are you not using it to download the file?

This comment has been minimized.

@jimhester

jimhester Jan 8, 2018

Member

I think the answer to my question is to auto-find the filename.

I guess it feels like this code should really be in httr, not here, maybe as a extension on httr::write_disk().

If we do keep it here, because we don't plan on a httr release in the near future, I think it would be better to write a write_disk() function and use it in a httr::GET() request.

This comment has been minimized.

@jennybc

jennybc Jan 8, 2018

Author Member

Doing this outside of httr, i.e. without access to unexported functions, doesn't look terribly practical. I think I can but it will definitely take me longer. How important is this @jimhester?

This comment has been minimized.

@jimhester

jimhester Jan 8, 2018

Member

Not that important, but how important is it to have this auto naming in usethis in the first place?

This comment has been minimized.

@jennybc

jennybc Jan 8, 2018

Author Member

Very important for this workshop use case!

I know it's hard to believe, but getting a large group of people to download a specific set of files to a folder on their computer that

a) has a sane name
b) they can find again
c) includes ALL the files and no other files

is astonishingly difficult. And I am going to get them to do it, so help me God. Or die trying.

R/course.R Outdated
hh <- curl::parse_headers_list(dl$headers)
stopifnot(hh[["content-type"]] == "application/zip")
content_disposition <- hh[["content-disposition"]]
stopifnot(!is.null(content_disposition), nzchar(content_disposition))

This comment has been minimized.

@jennybc

jennybc Jan 8, 2018

Author Member

Too rigid? A filename could potentially also be generated from either the input url or dl$url.

@jimhester

This comment has been minimized.

Copy link
Member

jimhester commented Jan 8, 2018

You can rename non-empty directories, I think the rename error is because the buzzy directory is not empty.

@jimhester

This comment has been minimized.

Copy link
Member

jimhester commented Jan 8, 2018

path_common() I don't think will help here, it finds the common prefix starting at the root, not a common stem anywhere.

@hadley

This comment has been minimized.

Copy link
Member

hadley commented Jan 8, 2018

You could split the paths into individual directories, and iteratively peel off common prefixes.

R/course.R Outdated
## I know I could use regex and lookahead but this is easier for me to
## maintain
if (grepl("^\"", cd) && grepl("\"$", cd)) {
cd <- gsub("^\"(.+)\"$", "\\1", cd)

This comment has been minimized.

@jimhester

jimhester Jan 9, 2018

Member

This should be sub() not gsub(), you are only doing at most one substitution per string, and the conditional is not needed, just use the substitution directly.

test <- function(cd) {
  sub("^\"(.+)\"$", "\\1", cd)
}
test("foo/bar")
#> [1] "foo/bar"
test('"foo/bar"')
#> [1] "foo/bar"
test('foo/"bar"')
#> [1] "foo/\"bar\""

Created on 2018-01-09 by the reprex package (v0.1.1.9000).

R/course.R Outdated
}
message("content-disposition:\n", cd)

cd <- gsub("^attachment;\\s*", "", cd, ignore.case = TRUE)

This comment has been minimized.

@jimhester

jimhester Jan 9, 2018

Member

This should be sub() it will only match (once) at the start of the string.

jennybc added some commits Jan 9, 2018

@jimhester

This comment has been minimized.

Copy link
Member

jimhester commented on 09cb6cd Jan 9, 2018

👍

@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Jan 9, 2018

Note to self: it would be really nice to show progress from download_zip().

@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Jan 9, 2018

@jimhester Do you agree it's not possible / easy for me to get progress, given I am using curl::curl_fetch_memory()? That's my assessment based on the C code.

jennybc added some commits Jan 9, 2018

@jennybc jennybc force-pushed the use-course branch from 09a7eea to 5cb4ffb Jan 10, 2018

@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Jan 10, 2018

I've made another major advance, so any comments are welcome. However, I'm not twiddling my thumbs. The usage example at the very top is has been updated for current state of things.

R/course.R Outdated
#' Special-purpose function to download a folder of course materials. The only
#' demand on the user is to confirm or specify where the new folder should be
#' stored. Workflow:
#' * User calls `use_course("SHORTLINK-GOES-HERE")`.

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

Shortlink = bit.ly?

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

bit.ly more obvious now

R/course.R Outdated
#' @param url Link to a ZIP file containing the materials, possibly behind a
#' shortlink. Function developed with DropBox and GitHub in mind, but should
#' work for ZIP files generally. See [download_zip()] for more.
#' @param destdir The new folder is stored here. Defaults to working directory.

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

Would it be better to default to the desktop? (Since everyone can find that)

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

Good point!

This comment has been minimized.

@jennybc

jennybc Jan 13, 2018

Author Member

Done d06440d. Usage examples at very top show this now.

R/course.R Outdated
#' work for ZIP files generally. See [download_zip()] for more.
#' @param destdir The new folder is stored here. Defaults to working directory.
#'
#' @return Path to the new directory holding the course materials.

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

Invisibly?

R/course.R Outdated
#' ## demo with a small CRAN package available in various places
#'
#' ## from CRAN
#' use_course("https://cran.r-project.org/bin/windows/contrib/3.4/rematch2_2.0.1.zip")

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

Needs example with shortlink too

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

I was reluctant to commit to this because I want the examples to work. But OK. I'll put a shortlink and make sure it works ... today.

#' ```
#' https://www.dropbox.com/sh/12345abcde/6789wxyz?dl=0
#' ```
#' Replace the `dl=0` at the end with `dl=1` to create a download link. The ZIP

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

Can we do this automatically?

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

Do you mean purely mechanically, i.e. on the front-end?

If yes: I could special-case DropBox and check the input url and make this substitution.

Otherwise, I don't think so. If input is a shortlink, we have no visibility into the redirects, to get at this URL and fix it. The failure mode is also non-specific (Error: Download does not have MIME type 'application/zip').

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

I have now learned this:

Information about any short bitly URL http://bit.ly/x is available at http://bit.ly/x+ (that is, the URL with a plus sign appended), for example http://bit.ly/1sNZMwL+. This allows users to see and check the long URL before visiting it.

It's not designed for programmatic use but may be helpful. In any case, probably not doing this now.

R/course.R Outdated
#' @return Path to the directory holding the unpacked files.
#' @keywords internal
#' @family download functions
#' @export

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

Again, might be better to wait?

R/course.R Outdated
keep <- function(file,
ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
ignores <- paste0("(\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z)")
!any(vapply(ignores, function(x) grepl(x, file, perl = TRUE), logical(1)))

This comment has been minimized.

@hadley

hadley Jan 12, 2018

Member

@jimhester is it worth having something like this in fs? Discarding paths matching globs is fairly common

This comment has been minimized.

@jimhester

jimhester Jan 12, 2018

Member

We could probably put something in there, we already do filtering in dir_ls(), so it wouldn't be too much work to wrap it in function.

This comment has been minimized.

@jimhester

jimhester Jan 12, 2018

Member

FWIW just added fs::path_filter() function r-lib/fs@195c8e4

R/course.R Outdated
}

keep <- function(file,
ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

Also, is my unzip-ignoring of .RData going to be an ugly surprise for anyone? It's the only here that I wondered about.

This comment has been minimized.

@jennybc

jennybc Jan 12, 2018

Author Member

At least, I've mentioned in the docs now.

R/course.R Outdated
invisible(target)
}

keep <- function(file,

This comment has been minimized.

@jimhester

jimhester Jan 12, 2018

Member

I think keep() could be rewritten somewhat more nicely.

In the following example keep_old is the current implementation, keep_lgl() is a tweaked form with the same return value. keep_re() simply returns the files to be kept rather than a logical vector and keep() uses fs::path_split() and does not use any regular expression. They all have equivalent output.

keep_old <- function(file,
                     ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {

  ignores <- paste0("(\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z)")
  !any(vapply(ignores, function(x) grepl(x, file, perl = TRUE), logical(1)))
}

keep_lgl <- function(file,
                     ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  ignores <- paste0("((\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z))", collapse = "|")
  !grepl(ignores, file, perl = TRUE)
}

library(testthat)
files <- c("foo", "bar", ".Rproj.user", ".git", "/.git", "/.git/", ".git/",
  "foo/.git", ".git", ".git/config", ".git/objects/06/3d3gysle", ".Rproj.user",
  ".Rproj.user/123jkl/persistent-state", ".Rhistory", ".RData", ".gitignore", "a/.gitignore", "foo.Rproj")
for (f in files) {
  expect_identical(keep_lgl(!!f), keep_old(!!f))
}

keep_re <- function(file,
                    ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  ignores <- paste0("((\\/|\\A)", gsub("\\.", "[.]", ignores), "(\\/|\\Z))", collapse = "|")
  grep(ignores, file, perl = TRUE, value = TRUE, invert = TRUE)
}
keep_re(files)
#> [1] "foo"          "bar"          ".gitignore"   "a/.gitignore"
#> [5] "foo.Rproj"
expect_equal(keep_re(files), files[vapply(files, keep_lgl, logical(1))])

keep <- function(file,
                 ignores = c(".Rproj.user", ".rproj.user", ".Rhistory", ".RData", ".git")) {
  file[vapply(fs::path_split(file), function(p) !any(p %in% ignores), logical(1))]
}

keep(files)
#> [1] "foo"          "bar"          ".gitignore"   "a/.gitignore"
#> [5] "foo.Rproj"
expect_equal(keep(files), keep_re(files))

Created on 2018-01-12 by the reprex package (v0.1.1.9000).

This comment has been minimized.

@jennybc

jennybc Jan 13, 2018

Author Member

Thanks, adopted in 5275eff.

jennybc added some commits Jan 12, 2018

Use @jimhester's improved keep function
The logical version is more useful to me in development/inspection.
@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Jan 13, 2018

I think I've implemented all the feedback. Usage example updated to reflect new default to ~/Desktop/ and usage with a bit.ly shortlink.

jennybc added some commits Jan 13, 2018

@hadley

This comment has been minimized.

Copy link
Member

hadley commented Jan 13, 2018

I think you should feel free to merge at this point; I'm sure we'll discover problems in the future, but this seems like a substantial amount of useful work.

@jennybc jennybc merged commit a62c56e into master Jan 13, 2018

5 of 6 checks passed

codecov/patch 55.79% of diff hit (target 63.12%)
Details
codecov/project 62.5% (-0.62%) compared to 390e05b
Details
continuous-integration/appveyor/branch AppVeyor build succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@jennybc jennybc deleted the use-course branch Jan 14, 2018

@jaredlander

This comment has been minimized.

Copy link

jaredlander commented Feb 9, 2018

Starting to use this now. First public attempt next week. Can we follow the download with a call to rstudioapi::openProject? That would make it easier for students rather than having to figure out how to open the project themselves. I would simply make it the next line in the script, but the destdir is unknown if they decide to change it.

@jaredlander

This comment has been minimized.

Copy link

jaredlander commented Feb 9, 2018

Actually, just realized I can assign projName <- use_course(...) then rstudio::openProject(projName). Might still be a nice convenient functionality for users.

@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Feb 9, 2018

We made a conscious choice to NOT create and launch an RStudio Project with use_course().

But I could be convinced. And/or we could make another function in the create_*() family that is use_course() + create & launch RStudio Project.

Thoughts, @hadley?

We also learned at rstudio::conf that hitting a GitHub repo with use_course() may be superior to DropBox. We had several people whose locked-down corporate laptops had trouble with DropBox but worked fine with GitHub. That's just a usage tip.

Also, a couple people needed to update the curl and backports packages, so obviously I need to determine and enforce minimum versions of those dependencies.

@jaredlander

This comment has been minimized.

Copy link

jaredlander commented Feb 10, 2018

I built a repo for a course I'm teaching. The README has them install some packages, run use_course, open the project and then source a script that came with the repo to download all the data. https://github.com/jaredlander/LiveFebruary2018

@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Feb 10, 2018

Looks good! I'd love to get feedback on how it works for you. Just so you know, you can use a shortlink with use_course(), if they're going to have to type it and you want to make that less error-prone.

@hadley

This comment has been minimized.

Copy link
Member

hadley commented Feb 10, 2018

Maybe we should automatically open a .Rproj file if there's one in the root directory?

@jaredlander

This comment has been minimized.

Copy link

jaredlander commented Feb 21, 2018

Used it last week and it was HUGE improvement over other ways I've tried to set up a class. Everyone had a project with the right structure so we no longer had path issues.

@jennybc

This comment has been minimized.

Copy link
Member Author

jennybc commented Feb 21, 2018

Used it last week and it was HUGE improvement over other ways I've tried to set up a class. Everyone had a project with the right structure so we no longer had path issues.

This brings me actual joy 😃, thanks for letting me know! The dev version now opens the folder as an RStudio project if you decided to include an .Rproj file.

@jaredlander

This comment has been minimized.

Copy link

jaredlander commented May 7, 2018

I built a package to generate a repo skeleton. This way instructors can easily create a new repo with a few lines of code. Then the user can go and do use_course on that generated repo.

This package creates the project on disk, populates the README with the desired packages and builds the file that students source to download data. Then it pushes all of that to GitHub so it is easily accessed by students.

Please see what you think: https://github.com/jaredlander/RepoGenerator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.