Add possibility to install from local Bioc mirror #31

jorainer · 2018-11-08T11:15:23Z

Would it be possible to add the possibility to install Bioconductor from a local mirror? With BiocInstaller and biocLite.R it was possible to specify the mirror with options(BioC_mirror = "<url to mirror>") but that does no longer work...

The text was updated successfully, but these errors were encountered:

mtmorgan · 2018-11-08T12:00:11Z

Let's have the conversation here rather than the bioc-devel mailing list, where I said

I'm not sure why this does not work.

The key is BiocManager::repositories(), which determines the repositories where R looks for packages. R choose the first repository where it finds the package (maybe of equivalent version, not exactly sure...). The first argument to repositories() can be a site repository, so

BiocManager::repositories("https://ima.repo")
                                        site_repository
                                     "https://ima.repo"
                                               BioCsoft
           "https://bioconductor.org/packages/3.9/bioc"
...

The Bioc repositories themselves are based on getOption("BioC_mirror") and getOption("repos"), so

options(BioC_mirror = "https://a.mirror.org")
BiocManager::repositories()
                                           BioCsoft
           "https://a.mirror.org/packages/3.9/bioc"
                                            BioCann
"https://a.mirror.org/packages/3.9/data/annotation"
...

These can be file paths; we are just delegating to install.packages().

It sounds like this doesn't work, somehow for you. Can you elaborate?

jorainer · 2018-11-08T12:17:25Z

Thanks @mtmorgan for the quick feedback.

It does indeed work on my notebook

options(BioC_mirror = "file:///data/shared/bioinf/mirror/bioconductor")
BiocManager::repositories()
                                                                     BioCsoft 
           "file:///data/shared/bioinf/mirror/bioconductor/packages/3.9/bioc" 
                                                                      BioCann 
"file:///data/shared/bioinf/mirror/bioconductor/packages/3.9/data/annotation" 
                                                                      BioCexp 
"file:///data/shared/bioinf/mirror/bioconductor/packages/3.9/data/experiment" 
                                                                BioCworkflows 
      "file:///data/shared/bioinf/mirror/bioconductor/packages/3.9/workflows" 
                                                                         CRAN 
                                                "https://cloud.r-project.org" 

sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin18.2.0/x86_64 (64-bit)
Running under: macOS  10.14.2

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BiocManager_1.30.3

loaded via a namespace (and not attached):
[1] compiler_3.5.1 tools_3.5.1

but not on the cluster:

> options(BioC_mirror = "file:///data/shared/bioinf/mirror/bioconductor")
> BiocManager::repositories()
Error: invalid version specification ‘Bioconductor version cannot be validated; no internet connection?’
> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid

Matrix products: default
BLAS: /shared/bioinf/R/R-3.5.1-BioC3.8/lib/R/lib/libRblas.so
LAPACK: /shared/bioinf/R/R-3.5.1-BioC3.8/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] BiocManager_1.30.3 compiler_3.5.1     tools_3.5.1

It does also not help if I specify the version with BiocManager::repositories(version = "3.8"). Seems a working internet connection is always required - which I don't have on the cluster. Also options(BIOCINSTALLER_ONLINE_DCF = FALSE) does not help (that is probably also only picked up by BiocInstaller).

Would be nice to have an option were internet connectivity is not required, e.g. if the version is specified...

mtmorgan · 2018-11-08T12:23:12Z

OK, issue #32 biting us again! Does it help to use a system variable or option

options(BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS=FALSE)

jorainer · 2018-11-08T12:27:58Z

Unfortunately not:

> options(BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS=FALSE)
> options(BioC_mirror = "file:///data/shared/bioinf/mirror/bioconductor")
> BiocManager::repositories()
Error: invalid version specification ‘Bioconductor version cannot be validated; no internet connection?’
> BiocManager::repositories(version = "3.8")
Error: Bioconductor version cannot be validated; no internet connection?
In addition: Warning message:
Bioconductor online version validation disabled; see
  ?BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS

onealdw · 2018-11-30T15:33:47Z

I am having the exact same issues as jotseung. rsync'd the repository according to https://bioconductor.org/about/mirrors/mirror-how-to/. The server in question is behind a firewall with no possibility of either opening the firewall port or setting up a http/https proxy. Any ideas when a work-around could be put into place to avoid requiring an internet connection>

mtmorgan · 2018-11-30T15:45:54Z

I think the work-around is to manually install BiocVersion from the appropriate Bioconductor repository, e.g., for Bioc 3.8 do the equivalent of

install.packages("BiocVersion", repos="https://bioconductor.org/packages/3.8/bioc")

Then when BiocManager tries to validate version, it'll find that it doesn't have internet access but it does at least know what version is in use, and will go with that.

onealdw · 2018-11-30T17:23:34Z

That didn't seem to help. I installed BiocVersion from the tar.gz source but I still get

    > BiocManager::install(version="3.8")
    Error: Bioconductor version cannot be validated; no internet connection?
    In addition: Warning message:
    Bioconductor online version validation disabled; see
      ?BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS

LiNk-NY · 2018-11-30T17:33:11Z

Hi @onealdw,
What version of BiocManager are you using?

onealdw · 2018-11-30T17:39:08Z

1.30.4

onealdw · 2018-12-02T16:50:09Z

Somewhat long script but I erased what I had already installed and tried from scratch. Again, this server is behind a firewall with no internet access. Also, I am a sysadmin, not a R expert, so please bear with me as I muddle my way through this. FYI, the system is running CentOS 7.5.

[nasapps@fsitgl-head01p ~]$ export R_LIBS_USER=/mnt/nasapps/production/bioconductor/3.8/lib
[nasapps@fsitgl-head01p ~]$ ls -l ${R_LIBS_USER}
total 0
[nasapps@fsitgl-head01p ~]$ R

R version 3.5.0 (2018-04-23) -- "Joy in Playing"
<snip>
> install.packages("/opt/nasapps/home/BiocManager_1.30.4.tar.gz", repos=NULL, type="source")
Installing package into ‘/mnt/nasapps/production/bioconductor/3.8/lib’
(as ‘lib’ is unspecified)
* installing *source* package ‘BiocManager’ ...
** package ‘BiocManager’ successfully unpacked and MD5 sums checked
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package ‘BiocManager’
    finding HTML links ... done
    BiocManager-pkg                         html
    available                               html
    install                                 html
    repositories                            html
    valid                                   html
    version                                 html
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (BiocManager)
> install.packages("/scratch/cluster_tmp/onealdw/bioconductor/packages/3.8/bioc/src/contrib/BiocVersion_3.8.0.tar.gz", repos=NULL, type="source")
Installing package into ‘/mnt/nasapps/production/bioconductor/3.8/lib’
(as ‘lib’ is unspecified)
* installing *source* package ‘BiocVersion’ ...
** help
*** installing help indices
  converting help for package ‘BiocVersion’
    finding HTML links ... done
    BiocVersion-pkg                         html
** building package indices
** testing if installed package can be loaded
* DONE (BiocVersion)
> options(BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS=FALSE)
> options(repos="file:///scratch/cluster_tmp/onealdw/bioconductor/packages/3.8/bioc")
> options(BioC_mirror = "file:///scratch/cluster_tmp/onealdw/bioconductor/packages/3.8/bioc")
> BiocManager::install(version="3.8")
Error: Bioconductor version cannot be validated; no internet connection?
In addition: Warning message:
Bioconductor online version validation disabled; see
  ?BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS
> BiocManager::version()
[1] ‘3.8’
>

onealdw · 2018-12-04T17:53:44Z

A work-around that let me install locally is to download https://bioconductor.org/config.yaml to disk. Edit BiocManager/R/version.R to set config on line 58 to point to the downloaded file.

LiNk-NY · 2018-12-14T00:39:06Z

Hi all,
I've looked into this and it seems to be originating first from .version_bioc
(and possibly other calls in the stack).

Essentially, there is no way to figure out what version corresponds to "devel" and
"release" without the internet version-map lookup.

One alternative is to try to guess "devel" by doing as.numeric(.local_version()[, 2]) %% 2) but this formula doesn't always hold up and it would only work for the current release cycle.

Another alternative is to modify / hard-code .VERSION_MAP_SENTINEL to indicate "devel" and "release" versions by corresponding to tail(.version_map(), 3).
This will then produce the right version even without an internet connection.

What are your thoughts Martin? @mtmorgan

mtmorgan · 2018-12-17T21:47:38Z

So next iteration @jotsetung @onealdw on the branch https://github.com/Bioconductor/BiocManager/tree/offline-version where I think

Sys.setenv(BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS=FALSE) # or options()
library(BiocManager)
BiocManager::repositories()
BiocManager::install("BiocGenerics")

works when the BiocVersion package has been installed, and there is a local repository?

jorainer · 2018-12-18T14:26:37Z

Thanks @mtmorgan for the update! I just have to find a way to test that - took me some time to convince the system administrators to allow connections from the cluster to Bioconductor. I am afraid to tell them now to temporarily disable internet access again ...

jorainer · 2018-12-19T06:48:54Z

I can confirm that the version from the offline-version branch works! Thanks!!!

> options(BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS=FALSE)
> library(BiocManager)
Bioconductor version 3.8 (BiocManager 1.30.4.9001), ?BiocManager::install for
  help
Warning message:
Bioconductor online version validation disabled; see
  ?BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS 
> BiocManager::repositories()
                                                                 BioCsoft 
           "file:///Volumes/jodata/mirror/bioconductor/packages/3.8/bioc" 
                                                                  BioCann 
"file:///Volumes/jodata/mirror/bioconductor/packages/3.8/data/annotation" 
                                                                  BioCexp 
"file:///Volumes/jodata/mirror/bioconductor/packages/3.8/data/experiment" 
                                                            BioCworkflows 
      "file:///Volumes/jodata/mirror/bioconductor/packages/3.8/workflows" 
                                                                     CRAN 
                                    "file:///Volumes/jodata/mirror/CRAN/" 
> BiocManager::install("BiocGenerics")
Bioconductor version 3.8 (BiocManager 1.30.4.9001), R 3.5.1 (2018-07-02)
Installing package(s) 'BiocGenerics'
Warning: failed to download mirrors file (cannot open URL 'https://cran.r-project.org/CRAN_mirrors.csv'); using local file '/Users/jo/R/2018-11/R-3.5.1-Bioc3.8/lib/R/doc/CRAN_mirrors.csv'
Warning message:
In download.file(url, destfile = f, quiet = TRUE) :
  URL 'https://cran.r-project.org/CRAN_mirrors.csv': status was 'Couldn't resolve host name'
* installing *source* package ‘BiocGenerics’ ...
** R
** inst
** byte-compile and prepare package for lazy loading
Creating a new generic function for ‘append’ in package ‘BiocGenerics’
Creating a new generic function for ‘as.data.frame’ in package ‘BiocGenerics’
Creating a new generic function for ‘cbind’ in package ‘BiocGenerics’
Creating a new generic function for ‘rbind’ in package ‘BiocGenerics’
...

vanhoan310 · 2019-03-12T13:42:48Z

I got the same problem because I use R on conda environment. I fixed it add file "cacert.pem" (in the anaconda environment) to R as follow.
Sys.setenv(CURL_CA_BUNDLE = "//path//to//cacert.pem file")
For details please see in here: http://tinyheero.github.io/2016/11/12/bioconductor-conda.html

ghost · 2019-03-15T11:33:07Z

Anyone knows where I can find offline-version?
This link https://github.com/Bioconductor/BiocManager/tree/offline-version is not working...

LiNk-NY · 2019-03-15T13:21:57Z

The offline-version branch should be merged to master. You can download it by getting the latest development version here (version 1.30.4.9002)

jorainer · 2019-05-10T06:33:29Z

Sorry to bug again, but with the new version it seems to still fail: I installed R-3.6.0, and BiocManager with install.packages("BiocManager"). Then I manually installed the BiocVersion_3.9.0.tar.gz package with R CMD INSTALL but BiocManager still refuses to install anything claiming that it has no internet access (which is in fact true - as our IT cut our cluster completely from the internet):

> options(BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS = FALSE)
> library(BiocManager())
Bioconductor version 3.9 (BiocManager 1.30.4), ?BiocManager::install for help
Bioconductor version cannot be validated; no internet connection?
Error: package or namespace load failed for ‘BiocManager’:
 .onAttach failed in attachNamespace() for 'BiocManager', details:
  call: NULL
  error: invalid version specification ‘Bioconductor version cannot be validated; no internet connection?’
In addition: Warning messages:
1: Bioconductor online version validation disabled; see
  ?BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS 
2: Bioconductor version cannot be validated; no internet connection? 

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux buster/sid

Matrix products: default
BLAS:   /shared/bioinf/R/R-3.6.0-BioC3.9/lib/R/lib/libRblas.so
LAPACK: /shared/bioinf/R/R-3.6.0-BioC3.9/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] BiocManager_1.30.4 compiler_3.6.0     tools_3.6.0

LiNk-NY · 2019-05-10T15:02:08Z

Hi Johannes, @jorainer
Thanks for reporting this. This error persists in the 1.30.4 version of BiocManager in CRAN.
We sent an update 1.30.5 to CRAN some time ago but the package has not been added to the
repository. I will contact them.

Best,
Marcel

jorainer · 2019-05-12T17:17:17Z

Thanks Marcel @LiNk-NY !

jorainer · 2019-05-14T10:04:49Z

I just installed the version manually and with 1.30.5 it works 👍

helterskelter01 · 2019-07-16T10:18:40Z

This has to be fixed soon. I got it installed by using @onealdw method.

I installed in an offline environemnt by doing the following,

Downloaded https://bioconductor.org/config.yaml to my local bioc server
Extracted BiocManager_1.30.4.tar.gz and changed BiocManager/R/version.R to point to my local bioc server.
Overwrote the BiocManager_1.30.4.tar.gz file in my local CRAN repo.
Installed BiocManager in R with: install.packages("BiocManager",repos='http://mycran.domain.com')
Installed BiocVersion in R with: install.packages('BiocVersion',repos='http://mybioc.domain.com/packages/3.9/bioc')
Installed stringi 1.4.4 with R CMD INSTALL
installed XML package in R.
Set options:
options(BioC_mirror='http://mybioc.domain.com')
options(repos='http://mycran.domain.com')
BiocManager::install()

LiNk-NY · 2019-07-16T16:46:27Z

Hi @helterskelter01

Thanks for your illustrative example. Offline installations are described
in the vignette for BiocManager version 1.30.5. We are working
with CRAN to figure out how their build system is set up so that
we can exactly replicate the errors that are manifested when
they build and check BiocManager.

Unfortunately, this whole process has been taking quite some time
(more than anyone would like).

Best regards,
Marcel

jorainer closed this as completed May 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add possibility to install from local Bioc mirror #31

Add possibility to install from local Bioc mirror #31

jorainer commented Nov 8, 2018

mtmorgan commented Nov 8, 2018

jorainer commented Nov 8, 2018

mtmorgan commented Nov 8, 2018

jorainer commented Nov 8, 2018

onealdw commented Nov 30, 2018

mtmorgan commented Nov 30, 2018

onealdw commented Nov 30, 2018

LiNk-NY commented Nov 30, 2018

onealdw commented Nov 30, 2018

onealdw commented Dec 2, 2018 •

edited

onealdw commented Dec 4, 2018

LiNk-NY commented Dec 14, 2018

mtmorgan commented Dec 17, 2018

jorainer commented Dec 18, 2018

jorainer commented Dec 19, 2018

vanhoan310 commented Mar 12, 2019

ghost commented Mar 15, 2019

LiNk-NY commented Mar 15, 2019

jorainer commented May 10, 2019

LiNk-NY commented May 10, 2019

jorainer commented May 12, 2019

jorainer commented May 14, 2019

helterskelter01 commented Jul 16, 2019

LiNk-NY commented Jul 16, 2019

Add possibility to install from local Bioc mirror #31

Add possibility to install from local Bioc mirror #31

Comments

jorainer commented Nov 8, 2018

mtmorgan commented Nov 8, 2018

jorainer commented Nov 8, 2018

mtmorgan commented Nov 8, 2018

jorainer commented Nov 8, 2018

onealdw commented Nov 30, 2018

mtmorgan commented Nov 30, 2018

onealdw commented Nov 30, 2018

LiNk-NY commented Nov 30, 2018

onealdw commented Nov 30, 2018

onealdw commented Dec 2, 2018 • edited

onealdw commented Dec 4, 2018

LiNk-NY commented Dec 14, 2018

mtmorgan commented Dec 17, 2018

jorainer commented Dec 18, 2018

jorainer commented Dec 19, 2018

vanhoan310 commented Mar 12, 2019

ghost commented Mar 15, 2019

LiNk-NY commented Mar 15, 2019

jorainer commented May 10, 2019

LiNk-NY commented May 10, 2019

jorainer commented May 12, 2019

jorainer commented May 14, 2019

helterskelter01 commented Jul 16, 2019

LiNk-NY commented Jul 16, 2019

onealdw commented Dec 2, 2018 •

edited