Can't install private bitbucket repos

[trinker]

I have used devtools::install_bitbucket to install private repos for my company for 3 years. Within the last 2 months the same code I used to use to install private repos no longger works. I can install public repos (so if I make the repo below that is private a public repo and then run remotes::install_bitbucket('trinker/cltext') eveything runs.

In reading the documentation I saw that it is likely that I need to set up an App Password. I set one up and copied it. I also saw that maybe I need to use the email rather than username in Googleing this. I tried this combo as well. All result in the same error message:

Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
cannot open URL 

In the app password I gave the instance full priveleges (checked every box possible).

MWE-ish

Code I'm trying showing all the different ways I've tried to run it. I know this is a private repo so this isn't really a MWE.

AppPassword = c('sdas_FAKE_APPKEY_23787', 'NewOne2:sdas_FAKE_APPKEY_23787', 'trinker:qVCTJqfLbM5JYg4VPKxK')
auth_user = c('trinker', 'tyler.rinker@gmail.com')


combos <- expand.grid(AppPassword, auth_user)

(calls <- sprintf("remotes::install_bitbucket('trinker/cltext', auth_user = '%s', password = '%s')", combos[,2], combos[,1]))

## [1] "remotes::install_bitbucket('trinker/cltext', auth_user = 'trinker', password = 'sdas_FAKE_APPKEY_23787')"                       
## [2] "remotes::install_bitbucket('trinker/cltext', auth_user = 'trinker', password = 'NewOne2:sdas_FAKE_APPKEY_23787')"               
## [3] "remotes::install_bitbucket('trinker/cltext', auth_user = 'trinker', password = 'trinker:qVCTJqfLbM5JYg4VPKxK')"               
## [4] "remotes::install_bitbucket('trinker/cltext', auth_user = 'tyler.rinker@gmail.com', password = 'sdas_FAKE_APPKEY_23787')"        
## [5] "remotes::install_bitbucket('trinker/cltext', auth_user = 'tyler.rinker@gmail.com', password = 'NewOne2:sdas_FAKE_APPKEY_23787')"
## [6] "remotes::install_bitbucket('trinker/cltext', auth_user = 'tyler.rinker@gmail.com', password = 'trinker:qVCTJqfLbM5JYg4VPKxK')"

attempts <- lapply(calls, function(x){ try(eval(parse(text = x))) })       


## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://trinker:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://trinker:NewOne2:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://trinker:trinker:qVCTJqfLbM5JYg4VPKxK@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://tyler.rinker@gmail.com:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://tyler.rinker@gmail.com:NewOne2:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://tyler.rinker@gmail.com:trinker:qVCTJqfLbM5JYg4VPKxK@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'

Session Info

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 16299)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] remotes_2.0.2.9000 pacman_0.5.0       forcats_0.3.0      stringr_1.3.1     
 [5] dplyr_0.7.8        purrr_0.2.5        readr_1.1.1        tidyr_0.8.2       
 [9] tibble_1.4.2       ggplot2_3.1.0      tidyverse_1.2.1   

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0       compiler_3.5.1   cellranger_1.1.0 pillar_1.3.0    
 [5] plyr_1.8.4       bindr_0.1.1      tools_3.5.1      jsonlite_1.5    
 [9] lubridate_1.7.4  gtable_0.2.0     nlme_3.1-137     lattice_0.20-38 
[13] pkgconfig_2.0.2  rlang_0.3.0.1    cli_1.0.1        rstudioapi_0.8  
[17] curl_3.2         haven_1.1.2      bindrcpp_0.2.2   withr_2.1.2     
[21] xml2_1.2.0       httr_1.3.1       hms_0.4.2        grid_3.5.1      
[25] tidyselect_0.2.5 glue_1.3.0       R6_2.3.0         readxl_1.1.0    
[29] modelr_0.1.2     magrittr_1.5     backports_1.1.2  scales_1.0.0    
[33] rvest_0.3.2      assertthat_0.2.0 colorspace_1.3-2 stringi_1.2.4   
[37] lazyeval_0.2.1   munsell_0.5.0    broom_0.5.0      crayon_1.3.4    

[gaborcsardi]

This works for me:

remotes::install_bitbucket("gaborcsardi/keyring", password="<app password here>")

The app password does not need full privileges at all, I only had "read repo", and apparently that is enough.

[trinker]

Hmm, I created a private repo and invited you to have read access: https://bitbucket.org/trinker/testprivate

Does it work for you to install this?

[gaborcsardi]

No, that does not work, unfortunately. I am not sure why, but seems like in this case BitBucket wants a proper Basic Auth in this case. Unfortunately base R cannot send an HTTP header. We'll fix this, but that will need the curl package.

[gaborcsardi]

So, after merging #256, this should be fixed.

install_bitbucket("trinker/testprivate", auth_user="gaborcsardi", password="<pass>")

works fine for me on macos. Hopefully #256 will work on Windows and Linux as well.

[trinker]

@gaborcsardi Thank you for taking a look and getting a fix so quick. This works on my end now.

[trinker]

One hiccup...from the command line I was fine because:

>  remotes:::download_method()
[1] "libcurl"

But in RStudio I am not because:

> remotes:::download_method()
[1] "wininet"

I think this may because RStudio is setting the following as the default option:

> getOption("download.file.method")
[1] "wininet"

Maybe because I have an outdated RStudio?

[gaborcsardi]

@trinker maybe you can set that to libcurl in your profile, and then RStudio will not override it?

[trinker]

Yes definitely. But not sure if RStudio should even be setting this.

[gaborcsardi]

Yeah, I agree, ideally it should not. I guess they did it, because the default causes (different) problems to many people. In particular, "wininet" uses windows's proxy config automatically. I think recent R versions select wininet automatically, anyway, but people might be using older ones.

Still, maybe "libcurl" is a better default nowadays, all the RStudio packages use the curl package anyway, and this requires the same proxy config as base's libcurl. I'll open an issue or talk to the RStudio IDE folks.

[gaborcsardi]

Yeah, for the IDE the problem is the proxies. If the default method is libcurl, then the proxy settings are not used on a new R installation, and you can't even use install.packages(). So I am afraid that the wininet default stays for now.

I'll try to submit a patch to R, that allows custom headers in download.file() and url(), for libcurl, and also for wininet.

[jimhester]

A patch in R to do this would be awesome