Use renv for private GitLab package on local server

[peteD900]

Hi there. Following up on a question I posted on Stack Overflow: I'd like to use renv to manage some packages hosted on a private local GitLab account. By local I mean hosted on a local server with GitLab on. I use RStudio Server Pro to develop packages and other users can install packages from GitLab by running:

devtools::install_git( url = "http://my-gitlab-server/r-projects/package.git", credentials = git2r::cred_user_pass("user", "pass") )

When I use renv::init() the package source is unknown and after help from Kevin Ushey (KU) the DESCRIPTION file of the package I'm trying to install as RemoteType: git2r. Following this I tried to use renv::install() to install the package after saving a GITLAB_PAT in a .Renviron file in the project root folder. I got the error:

download failed [error code 22]

I'm not sure how to use renv::install correctly in this scenario. KU suggested to try renv::install("gitlab::user/project") but I'm not sure this makes sense when the package is hosted on http://my-gitlab-server. I did try various combinations of strings to install:

• renv::install("gitlab::user/project")
• renv::install("gitlab::http://my-gitlab-server/r-projects/package.git")
• renv::install("gitlab::my-gitlab-server/r-projects/package.git")
• etc.

But all resulted in the same error and KU suggested raising an issue. Perhaps there is a better or different way to go about this?

sessionInfo()
R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
  [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8     LC_MONETARY=en_GB.UTF-8   
[6] LC_MESSAGES=en_GB.UTF-8    LC_PAPER=en_GB.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
  [1] stats     graphics  grDevices datasets  utils     methods   base     

loaded via a namespace (and not attached):
  [1] compiler_3.6.2 tools_3.6.2    renv_0.9.3 

[kevinushey]

I think something like the following should work:

renv::install("gitlab@my-gitlab-server::project/package")

Or, you can also try setting the gitlab.host renv option; e.g.

options(renv.config.gitlab.host = "my-gitlab-server")
renv::install("gitlab::project/package")

[fdetsch]

@peteD900 Also make sure that your GitLab PAT has the correct scope. I accidentally set one of mine to 'read_repository' and then stumbled across the same error. The code @kevinushey suggested now works fine after I created a new PAT with scope 'api'.

[peteD900]

Thanks @fdetsch but I still can't get this to work. I doubled checked the GITLAB_PAT and it has 'api' scope. I stored it in a .Renviron file and I can access the token via:

Sys.getenv("GITLAB_PAT")

"5hfhfiVNotreal7o3i2s65i"

I tried setting the GitLab host option:

options(renv.config.gitlab.host = "my-gitlab-server")

getOption("renv.config.gitlab.host")
[1] "my-gitlab-server"

Then running:
renv::install("gitlab::R-Projects/package")
and
renv::install("gitlab::R-Projects/package.git")
also
renv::install("gitlab@my-gitlab-server::R-Projects/package")
after unsetting the server option.

But I still get the error:

download failed [error code 7]
In addition: Warning messages:
1: In system2("curl", args$data(), stdout = TRUE, stderr = TRUE) :
running command ''curl' --config '/tmp/RtmpYGT7O1/renv-tempdir-aac9461255f7/renv-download-config-aac973f5c778' 2>&1' had status 7
2: In downloader(url, destfile, type, request, headers) :
curl: (7) Failed to connect to my-gitlab-server port 443: Connection refused

So I'm wondering whether it is just a connection issue at my end. But I'm confused as to why I can run:

devtools::install_git( "http://my-gitlab-server/R-Projects/package.git", credentials = git2r::cred_user_pass("user", "pass"))

In a project where I have not initialised renv. Any tips, tricks and hacks appreciated to figure this out. It's not a big deal but when another user runs renv::init() I have to help install the private packages separately.

[peteD900]

Ok got it.....not sure why I had to do this but if I set:
options(renv.config.gitlab.host = "http://my-gitlab-server")
instead of
options(renv.config.gitlab.host = "my-gitlab-server")
It worked. This does not seem to work:
renv::install("gitlab@my-gitlab-server::project/package") or
renv::install("gitlab@http://my-gitlab-server::project/package")

[dagola]

@peteD900 Also make sure that your GitLab PAT has the correct scope. I accidentally set one of mine to 'read_repository' and then stumbled across the same error. The code @kevinushey suggested now works fine after I created a new PAT with scope 'api'.

I ran in the same issue today and I'm wondering if a PAT with full 'api' scope is really necessary.

The situation at our place is that we have a dependency on a private repository during our build process using GitLab's CI/CD. At the moment we have to supply a PAT with full 'api' scope within the repository to enable the CI/CD job to access the other repository. Maybe this setting is dump at all but that's how it is for now.

Anyways, I think I tracked this issue down to this code block

renv/R/remotes.R

Lines 382 to 390 in 7fdff02

	fmt <- "%s/api/v4/projects/%s/repository/commits/%s"
	origin <- renv_retrieve_origin(host)
	id <- URLencode(paste(user, repo, sep = "/"), reserved = TRUE)
	url <- sprintf(fmt, origin, id, ref)
	destfile <- renv_tempfile("renv-gitlab-commits-")
	download(url, destfile = destfile, type = "gitlab", quiet = TRUE)
	json <- renv_json_read(file = destfile)
	sha <- json$id

And in there the call

renv/R/download.R

Lines 52 to 58 in 7101c91

	size <- renv_download_size(url, type, headers)
	if (size != -1 && file.exists(destfile)) {
	if (file.size(destfile) == size) {
	vwritef("\tOK [file is up to date]")
	return(destfile)
	}
	}

to query the size of the json with curl trying to access the header is the only point at which a PAT with full 'api' scope is needed. Interestingly, querying the file size within in this code block

renv/R/remotes.R

Lines 392 to 396 in 7fdff02

	# retrieve DESCRIPTION file
	fmt <- "%s/api/v4/projects/%s/repository/files/%s/raw?ref=%s"
	origin <- renv_retrieve_origin(host)
	id <- URLencode(paste(user, repo, sep = "/"), reserved = TRUE)
	url <- sprintf(fmt, origin, id, descpath, ref)

does not need a PAT with full 'api' scope. Maybe this is a bug on GitLab's side but maybe @kevinushey can think of a way to bypass this on renvs side?

[kevinushey]

We could skip that download size check when using GitLab -- would that be appropriate?

[dagola]

I'm not sure of any side effects skipping that check but if you think that's a solution, I'm very fine with it.

[pokyah]

Hi all,

I'm experiencing a similar problem.

I want renv to install a package from our company gitlab instance but with no success.

I have created a personal access token with API being the only option checked.

Then in my R project I execute the following :

  options(renv.config.gitlab.host = "https://git.mycompany.com")
  Sys.setenv(GITLAB_PAT = "JK_dnpbQ1NYd5Mcr_wEQ")
  renv::install("gitlab::myusername/mypackage")

And R returns me the following :

Erreur : failed to resolve remote 'gitlab::myusername/mypackage' -- failed to retrieve 'https://git.mycompany.com/api/v4/projects/myusername%2Fmypackage/repository/commits/master' [error code 22]
De plus : Warning message:
curl: (22) The requested URL returned error: 404 Not Found 
Traceback (most recent calls last):
13: renv::install("gitlab::myusername/mypackage")
12: lapply(remotes, renv_remotes_resolve)
11: FUN(X[[i]], ...)
10: withCallingHandlers(renv_remotes_resolve_impl(entry, latest), 
        error = error)
 9: renv_remotes_resolve_impl(entry, latest)
 8: renv_remotes_resolve_gitlab(parsed)
 7: download(url, destfile = destfile, type = "gitlab", quiet = TRUE)
 6: renv_download_error(url, "error code %i", status)
 5: stopf("failed to retrieve '%s' [%s]", url, msg, call. = FALSE)
 4: stop(sprintf(fmt, ...), call. = call.)
 3: .handleSimpleError(function (e) 
    {
        ...
    }, "failed to retrieve 'https://git.mycompany.com/api/v4/projects/myusername%2Fmypackage/repository/commits/master' [error code 22]", 
        base::quote(NULL))
 2: h(simpleError(msg, call))
 1: stop(simpleError(message = message, call = e$call))

But when I try to install using devtools :

devtools::install_git( 
  url = "http://git.mycompany.com/myusername/mypackage",
  credentials = git2r::cred_user_pass("myusername", "mypass")
)

The installation works.

I really need renv to work because I'm building a solution that calls renv programmatically to automatically configure projects environments for my colleagues in such a way that reproducibility will always be insured.

(Please note that I have edited the real domain name of our company gitlab instance, username and package name)

Also, we currently have no SSH access enabled on our gitlab instance (and I don't have the hand on this restriction). We always use HTTP to work with our gitlab instance)

Any idea of what could be wrong ?

Thanks for your precious help

[kevinushey]

You could also set options(renv.download.trace = TRUE) and see if you get any extra useful diagnostic information from renv. In particular, you should see the actual request being processed by curl -- are your authentication credentials being passed along as expected?

Does the PAT you're using have the requisite permissions needed to access things in your private repository? Or would you instead prefer to use username + password authentication, as in your git example?

[pokyah]

Hi @kevinushey,

Just have tried the following :

  options(renv.config.gitlab.host = "https://git.mycompany.com")
  options(renv.download.trace = TRUE)
  Sys.setenv(GITLAB_PAT = "JK_dnpbQ1NYd5Mcr_wEQ")
  renv::install("gitlab::myusername/mypackage")

R returns me this message :

  # Downloading 'https://cran.rstudio.com/src/contrib/PACKAGES.rds' [curl] =====

## Request ===================================================================
location
fail
silent
show-error
user-agent = "RStudio Desktop (1.4.1106); R (4.0.1 x86_64-w64-mingw32 x86_64 mingw32)"
url = "https://cran.rstudio.com/src/contrib/PACKAGES.rds"
output = "C:/Users/tgoossens/AppData/Local/Temp/RtmpkRVoMq/renv-35a09ae6e66"
connect-timeout = "20"
retry = "3"

## Output ====================================================================
[no output generated]

## Status ====================================================================
0

# Downloading 'https://cran.rstudio.com/bin/windows/contrib/4.0/PACKAGES.rds' [curl]

## Request ===================================================================
location
fail
silent
show-error
user-agent = "RStudio Desktop (1.4.1106); R (4.0.1 x86_64-w64-mingw32 x86_64 mingw32)"
url = "https://cran.rstudio.com/bin/windows/contrib/4.0/PACKAGES.rds"
output = "C:/Users/tgoossens/AppData/Local/Temp/RtmpkRVoMq/renv-35a03d85308f"
connect-timeout = "20"
retry = "3"

## Output ====================================================================
[no output generated]

## Status ====================================================================
0

# Downloading 'https://git.environnement.brussels/api/v4/projects/tgoossens%2FresumebenR/repository/commits/master' [curl]

## Request ===================================================================
location
fail
silent
show-error
user-agent = "RStudio Desktop (1.4.1106); R (4.0.1 x86_64-w64-mingw32 x86_64 mingw32)"
url = "https://git.mycompany.com/api/v4/projects/mysusername%2Fmypackage/repository/commits/master"
output = "C:/Users/myusername/AppData/Local/Temp/RtmpkRVoMq/renv-tempdir-35a042d6d7b/renv-35a04237683b"
connect-timeout = "20"
retry = "3"
header = "Private-Token: JK_dnpbQ1NYd5Mcr_wEQ"

The PAT only has the API right. Should I allow other options? If I can work with a username and password from renv that's even better as it will be easier to use for my colleagues that are not developers and not familiar with gitlab, PAT, etc.

At the moment my script that automatically configure projects with pre-installed custom package circumvents this problem by doing:

  setwd(path) # path = the path of the project to create programmatically
  renv::activate() # activate renv
  devtools::install_git(
    pkg_url,
    credentials = git2r::cred_user_pass(
      username = as.list(Sys.info())$login,
      password = pwd) # use devtools to install 
  )

  renv::snapshot(project = path, type = "all", prompt = FALSE)

which results that the package is installed into the project private library with devtools.

[kevinushey]

If I understand correctly, having the api scope should suffice... the only thing I can imagine that would be different here is that install_git() will first clone the repository (using e.g. git clone, or an equivalent provided by git2r I believe) and so perhaps the authentication workflows differ between that and the method that renv uses (which explicitly queries the Gitlab APIs)

[pokyah]

Hi @kevinushey,

I have found that I can also use a PAT with devtools.

Now I end u pusing this workflow:

setwd(path) # path = the path of the project to create programmatically
renv::activate() # activate renv
devtools::install_git(
  pkg_url,
  credentials = usethis::git_credentials(protocol = "https", auth_token = Sys.getenv("gitlab_token")))

renv::snapshot(project = path, type = "all", prompt = FALSE)

Hope it may help someone :)