Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't install private bitbucket repos #255

Closed
trinker opened this issue Nov 19, 2018 · 31 comments · Fixed by #256
Closed

Can't install private bitbucket repos #255

trinker opened this issue Nov 19, 2018 · 31 comments · Fixed by #256

Comments

@trinker
Copy link

trinker commented Nov 19, 2018

I have used devtools::install_bitbucket to install private repos for my company for 3 years. Within the last 2 months the same code I used to use to install private repos no longger works. I can install public repos (so if I make the repo below that is private a public repo and then run remotes::install_bitbucket('trinker/cltext') eveything runs.

In reading the documentation I saw that it is likely that I need to set up an App Password. I set one up and copied it. I also saw that maybe I need to use the email rather than username in Googleing this. I tried this combo as well. All result in the same error message:

Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
cannot open URL 

In the app password I gave the instance full priveleges (checked every box possible).

MWE-ish

Code I'm trying showing all the different ways I've tried to run it. I know this is a private repo so this isn't really a MWE.

AppPassword = c('sdas_FAKE_APPKEY_23787', 'NewOne2:sdas_FAKE_APPKEY_23787', 'trinker:qVCTJqfLbM5JYg4VPKxK')
auth_user = c('trinker', 'tyler.rinker@gmail.com')


combos <- expand.grid(AppPassword, auth_user)

(calls <- sprintf("remotes::install_bitbucket('trinker/cltext', auth_user = '%s', password = '%s')", combos[,2], combos[,1]))

## [1] "remotes::install_bitbucket('trinker/cltext', auth_user = 'trinker', password = 'sdas_FAKE_APPKEY_23787')"                       
## [2] "remotes::install_bitbucket('trinker/cltext', auth_user = 'trinker', password = 'NewOne2:sdas_FAKE_APPKEY_23787')"               
## [3] "remotes::install_bitbucket('trinker/cltext', auth_user = 'trinker', password = 'trinker:qVCTJqfLbM5JYg4VPKxK')"               
## [4] "remotes::install_bitbucket('trinker/cltext', auth_user = 'tyler.rinker@gmail.com', password = 'sdas_FAKE_APPKEY_23787')"        
## [5] "remotes::install_bitbucket('trinker/cltext', auth_user = 'tyler.rinker@gmail.com', password = 'NewOne2:sdas_FAKE_APPKEY_23787')"
## [6] "remotes::install_bitbucket('trinker/cltext', auth_user = 'tyler.rinker@gmail.com', password = 'trinker:qVCTJqfLbM5JYg4VPKxK')"

attempts <- lapply(calls, function(x){ try(eval(parse(text = x))) })       


## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://trinker:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://trinker:NewOne2:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://trinker:trinker:qVCTJqfLbM5JYg4VPKxK@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://tyler.rinker@gmail.com:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://tyler.rinker@gmail.com:NewOne2:sdas_FAKE_APPKEY_23787@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'
## Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
##   cannot open URL 'https://tyler.rinker@gmail.com:trinker:qVCTJqfLbM5JYg4VPKxK@api.bitbucket.org/2.0/repositories/trinker/cltext/src/master/DESCRIPTION'

Session Info

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 16299)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] remotes_2.0.2.9000 pacman_0.5.0       forcats_0.3.0      stringr_1.3.1     
 [5] dplyr_0.7.8        purrr_0.2.5        readr_1.1.1        tidyr_0.8.2       
 [9] tibble_1.4.2       ggplot2_3.1.0      tidyverse_1.2.1   

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0       compiler_3.5.1   cellranger_1.1.0 pillar_1.3.0    
 [5] plyr_1.8.4       bindr_0.1.1      tools_3.5.1      jsonlite_1.5    
 [9] lubridate_1.7.4  gtable_0.2.0     nlme_3.1-137     lattice_0.20-38 
[13] pkgconfig_2.0.2  rlang_0.3.0.1    cli_1.0.1        rstudioapi_0.8  
[17] curl_3.2         haven_1.1.2      bindrcpp_0.2.2   withr_2.1.2     
[21] xml2_1.2.0       httr_1.3.1       hms_0.4.2        grid_3.5.1      
[25] tidyselect_0.2.5 glue_1.3.0       R6_2.3.0         readxl_1.1.0    
[29] modelr_0.1.2     magrittr_1.5     backports_1.1.2  scales_1.0.0    
[33] rvest_0.3.2      assertthat_0.2.0 colorspace_1.3-2 stringi_1.2.4   
[37] lazyeval_0.2.1   munsell_0.5.0    broom_0.5.0      crayon_1.3.4    
@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 19, 2018

This works for me:

remotes::install_bitbucket("gaborcsardi/keyring", password="<app password here>")

The app password does not need full privileges at all, I only had "read repo", and apparently that is enough.

@trinker
Copy link
Author

trinker commented Nov 19, 2018

Hmm, I created a private repo and invited you to have read access: https://bitbucket.org/trinker/testprivate

Does it work for you to install this?

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 19, 2018

No, that does not work, unfortunately. I am not sure why, but seems like in this case BitBucket wants a proper Basic Auth in this case. Unfortunately base R cannot send an HTTP header. We'll fix this, but that will need the curl package.

gaborcsardi added a commit that referenced this issue Nov 20, 2018
BitBucket does not support Basic Auth via URL encoding,
apparently. So we use HTTP headers instead. This is somewhat
tricky, because base R  download.file()) does not allow
that directly. We need to shadow an internal function in utils,
and also make use of a bugs in curl and the internal
download methods, that let you specify multiple HTTP headers
via the User-Agent option.

This will also fix #255.
@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

So, after merging #256, this should be fixed.

install_bitbucket("trinker/testprivate", auth_user="gaborcsardi", password="<pass>")

works fine for me on macos. Hopefully #256 will work on Windows and Linux as well.

@trinker
Copy link
Author

trinker commented Nov 20, 2018

@gaborcsardi Thank you for taking a look and getting a fix so quick. This works on my end now.

gaborcsardi added a commit that referenced this issue Nov 20, 2018
BitBucket does not support Basic Auth via URL encoding,
apparently. So we use HTTP headers instead. This is somewhat
tricky, because base R  download.file()) does not allow
that directly. We need to shadow an internal function in utils,
and also make use of a bugs in curl and the internal
download methods, that let you specify multiple HTTP headers
via the User-Agent option.

This will also fix #255.
@trinker
Copy link
Author

trinker commented Nov 20, 2018

One hiccup...from the command line I was fine because:

>  remotes:::download_method()
[1] "libcurl"

But in RStudio I am not because:

> remotes:::download_method()
[1] "wininet"

I think this may because RStudio is setting the following as the default option:

> getOption("download.file.method")
[1] "wininet"

Maybe because I have an outdated RStudio?

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

@trinker maybe you can set that to libcurl in your profile, and then RStudio will not override it?

@trinker
Copy link
Author

trinker commented Nov 20, 2018

Yes definitely. But not sure if RStudio should even be setting this.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

Yeah, I agree, ideally it should not. I guess they did it, because the default causes (different) problems to many people. In particular, "wininet" uses windows's proxy config automatically. I think recent R versions select wininet automatically, anyway, but people might be using older ones.

Still, maybe "libcurl" is a better default nowadays, all the RStudio packages use the curl package anyway, and this requires the same proxy config as base's libcurl. I'll open an issue or talk to the RStudio IDE folks.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

Yeah, for the IDE the problem is the proxies. If the default method is libcurl, then the proxy settings are not used on a new R installation, and you can't even use install.packages(). So I am afraid that the wininet default stays for now.

I'll try to submit a patch to R, that allows custom headers in download.file() and url(), for libcurl, and also for wininet.

@jimhester
Copy link
Member

jimhester commented Nov 20, 2018

A patch in R to do this would be awesome

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

@jimhester it is quite easy, now that I looked up the details. libcurl and wininet are easy, internal is probably easy as well, and I think we can safely ignore the curl, wget, etc. methods, because there you can already pass headers with extra options, and they should have never existed anyway.

@trinker
Copy link
Author

trinker commented Nov 20, 2018

Unfortunately, setting options(download.file.method = 'libcurl') does not work in the .Rprofile or .Renviron. It has to be done once the session is started. If you do it after RStudio starts then the download happens as expected.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

@trinker that is bad, unfortunately. We can work around it in remotes.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

So, I just tried to do

options(download.file.method = "libcurl")

in c:\Users\Gabor\Documents\.Rprofile and that actually worked. RStudio 1.2.1114.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 20, 2018

@trinker what is your RStudio version?

@trinker
Copy link
Author

trinker commented Nov 21, 2018

@gaborcsardi I was incorrect. That worked to put it in the .Rprofile in that location.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 21, 2018

OK, that's better. I think I'll just document it that for basic auth you'll need to select the libcurl method (and set proxies), on Windows, in RStudio.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 27, 2018

@jimhester I think we might still want to use the old, header-less method for GH, though, because it works, and the new method does not work with the RStudio default "wininet" download method.

Or, even better, we could just try the header-less method for GH, if "wininet" is selected via the option. I'll leave this issue open until this is fixed.

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 27, 2018

Actually, it seems that the GH token is still used in the URL, so that's good then, forget it.

Also, this issue is already closed... :)

@gaborcsardi
Copy link
Contributor

gaborcsardi commented Nov 27, 2018

OK, added a note to the doc: 2e3bf35

@rodo-nunez
Copy link

rodo-nunez commented Dec 12, 2018

Hello!

I had the same problem. As this was very recent, I didn’t find this fix on 2.0.2 (last release), so I installed the github version (2.0.2.9000) and confirmed that this change was there on NEWS.md:

remotes dev

After this, I tried to install my private repository as I have been doing for a long time, but I got the same error. If I used devtools::install_bitbucket version 1.13.5 it worked, but with 2.0.0 or greater, it doesn’t (for devtools and remotes). This is my code

remotes::install_bitbucket("user/repo", auth_user = "user", password = "pass")

I tried running options(download.file.method = "libcurl") first, but I get the same error.

Am I forgetting to do something?

@trinker
Copy link
Author

trinker commented Dec 13, 2018

@rodo-nunez session info and the error message may be helpful here.

@rodo-nunez
Copy link

rodo-nunez commented Dec 13, 2018

Sorry, the error is:

Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'https://api.bitbucket.org/2.0/repositories/bitbucketteam/repo/src/master/DESCRIPTION'

My session info is

> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=Spanish_Chile.1252  LC_CTYPE=Spanish_Chile.1252   
[3] LC_MONETARY=Spanish_Chile.1252 LC_NUMERIC=C                  
[5] LC_TIME=Spanish_Chile.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.1     tools_3.4.1        yaml_2.1.19        remotes_2.0.2.9000

@N1KG
Copy link

N1KG commented Feb 6, 2019

Hi , I am experiencing the same problem. However, It does work when I set the repo public.

Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'https://api.bitbucket.org/2.0/repositories/bitbucketteam/repo/src/master/DESCRIPTION' 

My session info is

> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=English_Canada.1252  LC_CTYPE=English_Canada.1252    LC_MONETARY=English_Canada.1252
[4] LC_NUMERIC=C                    LC_TIME=English_Canada.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] remotes_2.0.2.9000 usethis_1.4.0      devtools_2.0.1    

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0        rstudioapi_0.9.0  magrittr_1.5      pkgload_1.0.2     R6_2.3.0          rlang_0.3.1      
 [7] tools_3.5.2       pkgbuild_1.0.2    sessioninfo_1.1.1 cli_1.0.1         withr_2.1.2       assertthat_0.2.0 
[13] digest_0.6.18     rprojroot_1.3-2   crayon_1.3.4      processx_3.2.1    callr_3.1.1       fs_1.2.6         
[19] ps_1.3.0          curl_3.3          memoise_1.1.0     glue_1.3.0        compiler_3.5.2    desc_1.2.0       
[25] backports_1.1.3   prettyunits_1.0.2

@wildintellect
Copy link

wildintellect commented Mar 4, 2019

Just hit this, no Windows machine I've tried works. Exact same repo, user, app password works fine on Linux. So just confirming it must be the download method. Doesn't matter if it's R directly or RStudio. Setting the option for libcurl in the session does not seem to work. Any reason the method arg isn't passed down to download.file?

Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.0        ps_1.3.0          prettyunits_1.0.2 rprojroot_1.3-2   digest_0.6.18     crayon_1.3.4      withr_2.1.2       assertthat_0.2.0  R6_2.4.0         
[10] backports_1.1.3   magrittr_1.5      rlang_0.3.1       cli_1.0.1         fs_1.2.6          remotes_2.0.2     callr_3.1.1       devtools_2.0.1    desc_1.2.0       
[19] tools_3.5.2       glue_1.3.0        pkgload_1.0.2     compiler_3.5.2    processx_3.2.1    pkgbuild_1.0.2    sessioninfo_1.1.1 memoise_1.1.0     usethis_1.4.0    

@wildintellect
Copy link

wildintellect commented Mar 5, 2019

Apologies, just tried the version of remotes in github and it does work if I change the options. However the new code does not pick libcurl even though it detected it (has_curl returns true, but download_method still uses winnet), and I had to force it by setting the options.

@wildintellect
Copy link

wildintellect commented Mar 5, 2019

A little more investigating, the download.file method has a default set and it's wininet.

> remotes:::has_curl()
[1] TRUE
> remotes:::download_method()
[1] "wininet"
> getOption("download.file.method")
[1] "wininet"

So this line in download_method() always has a value, and it always gets used even if it isn't good:
user_option <- getOption("download.file.method")

I think a better solution for now is to take the method args ... expansion. The docs say

... Other arguments passed on to utils::install.packages().

but that clearly isn't true, I think it would be more obvious than setting the session options.

@jakegalson
Copy link

jakegalson commented Mar 6, 2019

Hi all,

I am having the same issue. I have set the download option to libcurl:

options(download.file.method = 'libcurl')

And this still happens:

> remotes:::install_bitbucket("reporedacted", auth_user="user", password="pass")
Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'URLredacted'

Same command works on linux. Any help is much appreciated!

Session info is as follows:

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] kylib_0.1.1    usethis_1.4.0  devtools_2.0.1 RJDBC_0.2-5    rJava_0.9-8    DBI_0.6       

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.13      rstudioapi_0.8    magrittr_1.5      pkgload_1.0.2     debugme_1.1.0     R6_2.2.2          rlang_0.2.1      
 [8] stringr_1.2.0     plyr_1.8.4        tools_3.3.3       pkgbuild_1.0.2    config_0.3        sessioninfo_1.1.1 cli_1.0.0        
[15] withr_2.1.2       remotes_2.0.2     yaml_2.1.14       assertthat_0.2.0  digest_0.6.12     rprojroot_1.3-2   crayon_1.3.4     
[22] reshape2_1.4.2    callr_2.0.3       fs_1.2.2          testthat_1.0.2    glue_1.1.1        memoise_1.1.0     stringi_1.1.5    
[29] desc_1.2.0        backports_1.1.2   prettyunits_1.0.2

@wildintellect
Copy link

wildintellect commented Mar 14, 2019

@jakegalson did you upgrade remotes from github 1st? the CRAN version won't work with the override currently.

@spsanderson
Copy link

spsanderson commented May 30, 2019

I have the same error. I am trying to run the initial script for RemixAutoML in step 1 https://github.com/AdrianAntico/RemixAutoML

> sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] usethis_1.5.0  devtools_2.0.2

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.1        ps_1.3.0          prettyunits_1.0.2 rprojroot_1.3-2  
 [5] withr_2.1.2       digest_0.6.19     crayon_1.3.4      assertthat_0.2.1 
 [9] R6_2.4.0          backports_1.1.4   magrittr_1.5      rlang_0.3.4      
[13] cli_1.1.0         curl_3.3          fs_1.3.1          remotes_2.0.4    
[17] rstudioapi_0.10   callr_3.2.0       desc_1.2.0        tools_3.5.3      
[21] glue_1.3.1        pkgload_1.0.2     compiler_3.5.3    processx_3.3.1   
[25] pkgbuild_1.0.3    sessioninfo_1.1.1 memoise_1.1.0    
> remotes:::has_curl()
[1] TRUE
> remotes:::download_method()
[1] "libcurl"
> getOption("download.file.method")
[1] "libcurl"

> for (i in to_install) {
+   message(paste("looking for ", i))
+   if(i == "catboost" & !requireNamespace(i)) {
+     devtools::install_github('catbo .... [TRUNCATED] 
looking for  catboost
Loading required namespace: catboost
Failed with error:  ‘there is no package called ‘catboost’’
Downloading GitHub repo catboost/catboost@master
Error in utils::download.file(url, path, method = download_method(), quiet = quiet,  : 
  cannot open URL 'https://api.github.com/repos/catboost/catboost/tarball/master'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants