Add SHA updating logic for github remotes #135

jimhester · 2018-07-31T15:25:40Z

This updates remotes with the logic from the current devel devtools (only for the github remote).

Since pkgman and friends seem to be a ways off it might be worth trying to update remotes further and use that on travis / appveyor etc. The process was somewhat easier than I expected originally.

jimhester · 2018-07-31T15:26:28Z

R/github.R

+  tmp <- tempfile()
+  download(tmp, url, auth_token = github_pat())
+
+  rawToChar(base64enc::base64decode(gsub("\\\\n", "", fromJSONFile(tmp)$content)))


This is the current blocker for this, we need to decode the base64 encoded content, so I guess would have to write a R only version of base64decode().

Update: now have this

yihui · 2018-08-01T15:04:29Z

Yeah!! 🎉 🎉 🎉

codecov-io · 2018-08-02T16:33:11Z

Codecov Report

❗ No coverage uploaded for pull request base (master@e78d84e). Click here to learn what that means.
The diff coverage is 73.34%.

@@            Coverage Diff            @@
##             master     #135   +/-   ##
=========================================
  Coverage          ?   82.02%           
=========================================
  Files             ?       27           
  Lines             ?     1524           
  Branches          ?        0           
=========================================
  Hits              ?     1250           
  Misses            ?      274           
  Partials          ?        0

Impacted Files	Coverage Δ
R/json.R	`93.68% <0%> (ø)`
R/utils.R	`86.61% <100%> (ø)`
R/github.R	`27.58% <13.63%> (ø)`
R/install-github.R	`73.64% <14.28%> (ø)`
R/install-git.R	`20.73% <17.14%> (ø)`
R/install-local.R	`84% <60%> (ø)`
R/install-remote.R	`71.21% <65.45%> (ø)`
R/deps.R	`81.63% <67.67%> (ø)`
R/install-svn.R	`91.89% <86.84%> (ø)`
R/install-bioc.R	`89.68% <89.68%> (ø)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e78d84e...d82007f. Read the comment docs.

jimhester · 2018-08-06T14:38:16Z

@gaborcsardi I realize this is probably too big now to review in detail; do you have any overall comments and would you be ok with me merging this and working on the open issues?

gaborcsardi · 2018-08-06T15:25:58Z

Ill take a look later today.

…

On Mon, 6 Aug 2018, 16:38 Jim Hester, ***@***.***> wrote: @gaborcsardi <https://github.com/gaborcsardi> I realize this is probably too big now to review in detail; do you have any overall comments and would you be ok with me merging this and working on the open issues? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#135 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAoTQC1yHD6tgCX3yfh1RNyAVpRjzMNaks5uOFTZgaJpZM4Voe2D> .

gaborcsardi

Super cool! Left some minor comments. Thanks for doing this.

gaborcsardi · 2018-08-06T19:21:58Z

R/deps.R

+  }
+  fun <- tryCatch(get(paste0(tolower(type), "_remote"),
+      envir = asNamespace("remotes"), mode = "function", inherits = FALSE),
+    error = function(e) stop("Unknown remote type: ", type, call. = FALSE))


IDK, this will find things like parse_one_remote(), etc. Not that this happens in practice, but it does not seem right to me. How about just create an object, try to dispatch, and then fail, if the dispatch fails.

gaborcsardi · 2018-08-06T19:26:18Z

R/install-git.R

 #'}
-install_git <- function(url, subdir = NULL, branch = NULL,
+install_git <- function(url, subdir = NULL, ref = NULL,


This is a breaking change. Can we re-add a branch argument and forward that to ref? If you think it is worth it. Only in the user facing function.

gaborcsardi · 2018-08-06T19:31:37Z

R/install-git.R

+  on.exit(unlink(tmp))
+  description_path <- paste0(collapse = "/", c(remote$subdir, "DESCRIPTION"))
+
+  # Try using git archive --remote to retrieve the DESCRIPTION, if the protocol


Seems like this means that even for git2r remotes we need a command line git? So can't we just use that for everything and remove the git2r stuff?

You need command line git to get the remote package name, but if you can't get it because command line git is not available the thing we then just install the remote unconditionally. So it will still work without command line git, just won't be as efficient as it could be.

I think it is still useful to have both types of remotes, although I agree it does increase the complexity considerably; the code is already written...

gaborcsardi · 2018-08-06T19:32:47Z

R/install-git.R

+  res <- try(silent = TRUE,
+    system_check(git_path(),
+      args = c("archive", "-o", tmp, "--remote", remote$url,
+        if (is.null(remote$ref)) "HEAD" else remote$ref,


This is sometimes HEAD, sometimes master (see below). Which one is right?

Probably HEAD as it is most generic

gaborcsardi · 2018-08-06T19:55:34Z

R/install-gitlab.R

@@ -0,0 +1,138 @@
+#' Instlal a package from GitLab


gaborcsardi · 2018-08-06T19:57:30Z

R/install-remote.R

+    local_sha <- local_sha(remote_package_name(remote))
+  }
+
+  same <- remote_sha == local_sha


Do we always have a full sha here? (As opposed to the first 7 characters or sg like that?)

Yes, IIRC all of the methods return the full untruncated sha.

gaborcsardi · 2018-08-06T19:59:02Z

R/install-remote.R

+  package2remote(name)$sha %||% NA_character_
+}
+
+package2remote <- function(name, lib = .libPaths(), repos = getOption("repos"), type = getOption("pkgType")) {


This seems like an important internal function. Do you think you could add some brief (internal) docs that explain what it is used for? Thanks!

You are right, I wonder if it would be useful to export this actually, it can sometimes be handy, e.g. if you want to re-install a package from another library. Although this would also require exporting install_remote().

gaborcsardi · 2018-08-06T20:00:03Z

R/install-remote.R

+}
+
+#' @export
+`[.remotes` <- function(x,i,...) {


Why is this trick needed? Can you please add it as internal docs, or a comment?

It is possible it is not needed, I will remove it and see if anything breaks.

It was not installed, so coverage was not running on travis previously

This avoids a breaking change

This reverts commit 2d4c782.

This seems to be due to a regression in roxygen 6.1.0

Mainly to avoid duplicating the protocol if it exists

This is about ~4 times faster than without because we can return just the SHA by passing the appropriate header bench::mark(iterations=30, curl = github_commit("r-lib", "devtools", host = "api.github.com", use_curl = TRUE), base = github_commit("r-lib", "devtools", host = "api.github.com", use_curl = FALSE)) # A tibble: 2 x 14 expression min mean median max `itr/sec` <chr> <bch:tm> <bch:tm> <bch:tm> <bch:tm> <dbl> 1 curl 68.6ms 84.2ms 81.4ms 164ms 11.9 2 base 261.8ms 316.4ms 284.9ms 615ms 3.16

This was missing in the remote dependencies, which caused rbind's to fail

It has changed in R-devel

jimhester · 2018-08-08T14:53:31Z

Thank you for the review Gabor, I appreciated the comments!

jimhester commented Jul 31, 2018

View reviewed changes

jimhester requested a review from gaborcsardi July 31, 2018 17:34

jimhester added 5 commits August 2, 2018 10:29

Add SHA updating logic for github remotes

9d4d98d

Add support for native base64 decoding

04fe763

Avoid R CMD check notes about missing imports

cd10138

Use git to install bioc packages

bb2f557

Updates for git2r 0.23.0

cfe857d

jimhester force-pushed the devtools-github branch from 2170efa to 23f86b2 Compare August 2, 2018 14:30

jimhester force-pushed the devtools-github branch 2 times, most recently from 2d551e7 to fb58c49 Compare August 3, 2018 21:23

gaborcsardi approved these changes Aug 6, 2018

View reviewed changes

jimhester added 15 commits August 8, 2018 10:44

Use remotes to install remotes

f9d56da

We don't need latex

524dc6f

Tweak check

427bdb2

Add support for git remotes

c91f09e

Add support for bitbucket

8032939

Support for local remotes

59847b1

Add svn support

a6777ee

Fix tests when CRAN repo is not set to cran.rstudio.com

7363a90

Use the correct Bioc branch for the version of R being tested

66bc8a2

Add logic for updating remotes

fea9e13

Update bitbucket to use the bitbucket API

72819f4

Use build_url in github functions

49a2f7c

Re-document

60a9f40

Add covr to Suggests

2fd7d0b

It was not installed, so coverage was not running on travis previously

install_packages needs the package name, not the remote

9624606

jimhester added 16 commits August 8, 2018 10:44

Tweak error logic for remotes

d081ff7

Preserve branch as argument name for install_git

9cc50ce

This avoids a breaking change

Use HEAD for all cases

299d066

Fix typos

603ae36

Document package2remote a bit

e05f8d1

Remove un-needed method

462a428

Re-document with released roxygen 6.1.0

839f398

Do not use memoise

7d24982

Revert "Remove un-needed method"

d144559

This reverts commit 2d4c782.

Use rawNamespace to create the S3method registration

a44bb8c

This seems to be due to a regression in roxygen 6.1.0

Use build_url for github urls

9ee0b51

Mainly to avoid duplicating the protocol if it exists

Remove quiet argument

8863095

Add is_cran column

d572e88

This was missing in the remote dependencies, which caused rbind's to fail

Do not check for error message

40ad547

It has changed in R-devel

Remove un-neeeded method

d82007f

jimhester force-pushed the devtools-github branch from 61fe315 to d82007f Compare August 8, 2018 14:44

jimhester merged commit eb81fe6 into r-lib:master Aug 8, 2018

jimhester mentioned this pull request Aug 16, 2018

Improved install_bitbucket() interface and functionality #104

Closed

imanuelcostigan mentioned this pull request Jan 4, 2020

install_bitbucket() fails for privately hosted bitbucket server repositories. #429

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SHA updating logic for github remotes #135

Add SHA updating logic for github remotes #135

jimhester commented Jul 31, 2018

jimhester Jul 31, 2018

jimhester Jul 31, 2018

yihui commented Aug 1, 2018

codecov-io commented Aug 2, 2018 •

edited

Loading

jimhester commented Aug 6, 2018

gaborcsardi commented Aug 6, 2018 via email

gaborcsardi left a comment

gaborcsardi Aug 6, 2018

gaborcsardi Aug 6, 2018

gaborcsardi Aug 6, 2018

jimhester Aug 8, 2018

gaborcsardi Aug 6, 2018

jimhester Aug 8, 2018

gaborcsardi Aug 6, 2018

gaborcsardi Aug 6, 2018

jimhester Aug 8, 2018

gaborcsardi Aug 6, 2018

jimhester Aug 8, 2018

gaborcsardi Aug 6, 2018

jimhester Aug 8, 2018

jimhester commented Aug 8, 2018

Add SHA updating logic for github remotes #135

Add SHA updating logic for github remotes #135

Conversation

jimhester commented Jul 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihui commented Aug 1, 2018

codecov-io commented Aug 2, 2018 • edited Loading

Codecov Report

jimhester commented Aug 6, 2018

gaborcsardi commented Aug 6, 2018 via email

gaborcsardi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimhester commented Aug 8, 2018

codecov-io commented Aug 2, 2018 •

edited

Loading