Providing a fall-back mechanism to install an older version any package via devtools

[mmokrejs]

Hi,
I need to setup an old version of R-3.4.4 with some packages. To get devtools installed I am facing a dependency hell. Luckily, one can install sufficiently old version athough the README.md does not mention when the dependencies grew up over the years.

$ wget https://github.com/r-lib/devtools/archive/refs/tags/devtools-0.5.tar.gz
$ R CMD INSTALL devtools-0.5.tar.gz 
$ R -e 'install.packages("RCurl")'
$ wget https://github.com/r-lib/httr/archive/refs/tags/v1.4.2.tar.gz
$ R CMD INSTALL v1.4.2.tar.gz

If possible, a lazy-loading approach in devtools might be used so that one could use the package although with some restrictions. But I did not check the code.

I was hoping https://github.com/duckmayr/oldr will help me to automagically walk back any package version and any which get installed, if not, tryn an older one. Seems not.

[mmokrejs]

These are merely my notes of commands but the rlang package is a blocker for me. At the best 1.0.6 can be installed on R-3.4.4. Nevertheless, there are other people who are trying to get running an older version R in some container, so this might be useful for them:

wget https://github.com/r-lib/memoise/archive/refs/tags/v.1.1.0.tar.gz
mv v.1.1.0.tar.gz memoise-1.1.0.tar.gz
R CMD INSTALL memoise-1.1.0.tar.gz
wget https://cran.r-project.org/src/contrib/Archive/tibble/tibble_1.3.0.tar.gz
R CMD INSTALL tibble_1.3.0.tar.gz
wget https://github.com/r-lib/devtools/archive/refs/tags/devtools-0.5.tar.gz
R CMD INSTALL devtools-0.5.tar.gz
wget https://github.com/r-lib/devtools/archive/refs/tags/v1.4.tar.gz
mv v1.4.tar.gz devtools-1.4.tar.gz
R CMD INSTALL devtools-1.4.tar.gz
wget https://github.com/r-lib/devtools/archive/refs/tags/v1.11.1.tar.gz
mv v1.11.1.tar.gz devtools-1.11.1.tar.gz
R CMD INSTALL devtools-1.11.1.tar.gz
wget https://github.com/r-lib/devtools/archive/refs/tags/v1.13.6.tar.gz
mv  v1.13.6.tar.gz devtools-1.13.6.tar.gz
R CMD INSTALL devtools-1.13.6.tar.gz

apt-get -y build-dep libcurl4-gnutls-dev
apt-get -y install libcurl4-gnutls-dev

R CMD INSTALL devtools-1.13.6.tar.gz
wget https://github.com/r-lib/rlang/archive/refs/tags/v1.0.6.tar.gz
mv v1.0.6.tar.gz rlang-1.0.6.tar.gz
R CMD INSTALL rlang-1.0.6.tar.gz
wget https://github.com/r-lib/cachem/archive/refs/tags/v1.0.8.tar.gz
mv v1.0.8.tar.gz cachem-1.0.8.tar.gz
R CMD INSTALL cachem-1.0.8.tar.gz
wget https://github.com/r-lib/httr/archive/refs/tags/v1.4.4.tar.gz
mv v1.4.7.tar.gz httr-1.4.4.tar.gz
R CMD INSTALL httr-1.4.4.tar.gz

wget https://cran.r-project.org/src/contrib/kernlab_0.9-32.tar.gz
gzip -dc kernlab_0.9-32.tar.gz |  tar xvf -
wget https://github.com/cran/kernlab/commit/95702d49adc3dd1bb91667070ab57c2c49a1b933.patch
cd kernlab
# https://github.com/cran/kernlab/pull/3
patch -p1 < ../95702d49adc3dd1bb91667070ab57c2c49a1b933.patch
cd ..
tar cvf - kernlab | gzip -c > kernlab_0.9-32.tar.gz
R CMD INSTALL kernlab_0.9-32.tar.gz

R -e 'install.packages("lifecycle")'


wget https://github.com/r-lib/vctrs/archive/refs/tags/v0.5.2.tar.gz
mv v0.5.2.tar.gz vctrs-0.5.2.tar.gz
R CMD INSTALL vctrs-0.5.2.tar.gz

wget https://cran.r-project.org/src/contrib/Archive/locfit/locfit_1.5-9.2.tar.gz
R CMD INSTALL locfit_1.5-9.2.tar.gz
https://cran.r-project.org/src/contrib/Archive/cpp11/cpp11_0.4.5.tar.gz
R CMD INSTALL cpp11_0.4.5.tar.gz

wget https://cran.r-project.org/src/contrib/pillar_1.9.0.tar.gz
R CMD INSTALL pillar_1.9.0.tar.gz
wget https://cran.r-project.org/src/contrib/tibble_3.2.1.tar.gz
R CMD INSTALL tibble_3.2.1.tar.gz
wget https://cran.r-project.org/src/contrib/Archive/gtable/gtable_0.3.1.tar.gz
R CMD INSTALL gtable_0.3.1.tar.gz
wget https://cran.r-project.org/src/contrib/scales_1.2.1.tar.gz
R CMD INSTALL scales_1.2.1.tar.gz

I woud be happy if exactly these automated attempts to fetch older versions of packages were attempted by devtools automatically.

Something like the unfinished sketch of install_archive() function at https://stackoverflow.com/questions/16091304/efficiently-getting-older-versions-of-r-packages . It does not work for me, probably the parsing of available package versions is broken and hence it cannot create a list a walk it backwards, from newest to oldest.

[jennybc]

This seems like a general problem one could encounter with any package, yes? I don't see any specific devtools angle or solution here. Perhaps the archaelogical approach taken by the rang package could be helpful for this sort of unplanned time travel: https://github.com/gesistsa/rang.

[mmokrejs]

Yes, this would be helpful in many scenarios, for any packages. Basically, sometimes the latest and greatest have a new bug preventing installation but one can still install a previous one. devtools could not exit but instead, try a previous version, in a loop or could be configurable how many versions back it might try.

But this would also help finding by trial and error a package for some older R version which one needs to revive or in general, just to get something installed.

I felt that this would just need a loop per a list of versions per whatvever package to be acted upon and a switch not to exit upon an error. Anyway devtools already do try to autoresolve errors so I do not see this to be that much of a concern.

While having said that, I will try https://github.com/gesistsa/rang if that helps me out with yet another R instance and yet another environment. One also must be capable to handle the stuff mentally. Definitely seems like a rigorous approach, offering more than I even asked for. Thank you, @jennybc .

[mmokrejs]

> remotes::install_github("gesistsa/rang")
Downloading GitHub repo gesistsa/rang@HEAD
Skipping 3 packages not available: rlang, vctrs, httr
Installing 3 packages: here, renv, pkgsearch
trying URL 'https://cloud.r-project.org/src/contrib/here_1.0.1.tar.gz'
Content type 'application/x-gzip' length 32954 bytes (32 KB)
==================================================
downloaded 32 KB

trying URL 'https://cloud.r-project.org/src/contrib/renv_1.0.2.tar.gz'
Content type 'application/x-gzip' length 1135957 bytes (1.1 MB)
==================================================
downloaded 1.1 MB

trying URL 'https://cloud.r-project.org/src/contrib/pkgsearch_3.1.2.tar.gz'
Content type 'application/x-gzip' length 49766 bytes (48 KB)
==================================================
downloaded 48 KB

* installing *source* package ‘here’ ...
** package ‘here’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (here)
* installing *source* package ‘renv’ ...
** package ‘renv’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (renv)
* installing *source* package ‘pkgsearch’ ...
** package ‘pkgsearch’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (pkgsearch)

The downloaded source packages are in
	‘/tmp/Rtmpwg8LEp/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
── R CMD build ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/tmp/Rtmpwg8LEp/remotes32062ba95d0c/gesistsa-rang-411d539/DESCRIPTION’ (356ms)
─  preparing ‘rang’:
✔  checking DESCRIPTION meta-information ...
─  installing the package to process help pages
         -----------------------------------
   ERROR: this R is version 3.4.4, package 'rang' requires R >= 3.5.0
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'rang' from GitHub:
  ! System command 'R' failed
> 

This was quick.

[mmokrejs]

And under R-3.5.3 I anyway get with a demo command:

> library(rang)
> x <- resolve(pkgs = c("sna", "schochastics/rtoot", "S4Vectors"), snapshot_date = "2022-11-30")
Error in strsplit(a, "[.-]") : non-character argument
> 

and btw, devtools in tehir currenty version anyway cannot be installed because of

ERROR: dependency ‘usethis’ is not available for package ‘devtools’
* removing ‘/home/bionic/R/x86_64-pc-linux-gnu-library/3.5/devtools’

The downloaded source packages are in
	‘/tmp/RtmpEJ0LFd/downloaded_packages’
Warning message:
In install.packages("devtools") :
  installation of package ‘devtools’ had non-zero exit status
>

So I will again need to install manually some older version of devtools, usethis by trial and error ... to get the 3.5.3 environment at least partly working like 3.4.4 already was.

[jennybc]

Basically, sometimes the latest and greatest have a new bug preventing installation but one can still install a previous one. devtools could not exit but instead, try a previous version, in a loop or could be configurable how many versions back it might try.

I see where you're coming from and this has been wished for by others. It just turns out that this is actually a really hard problem, due to the way the dependency graph of R packages evolves over time, with all of their minimum version requirements (for other packages and R itself) changing constantly. If you just want to install the previous version of a package, such as devtools, that usually works without too much fuss. You're doing something quite different, when you try to work on R 3.4.4 (released 2018-03-15), which is not actually supported by any recent version of devtools or its dependencies.

This qualifies as serious time travel and, in that case, you really do need to capture a working environment explicitly in a docker container or similar or use some tool that helps you install everything as it was on a certain date.

[jennybc]

This looks like it might be another useful resource for planning for reproducibility (with big emphasis on docker, renv, etc.):

https://raps-with-r.dev/repro_intro.html

[mmokrejs]

Truly said, I started with some old docker image an author of a package prepared, but as it starts firefox from insuide the container one is supposed disable X11 security controls, etc. I tried to migrate it to LXC containers, umm, did not complete that. So I created an LXC container based on the same distro and version as was the docker image, hence Bionic 18.04 with R-3.4.4 and Bioconductor-3.6. It is 10 working days behind.

This seems to work for me although I will need to check which packages are still failing.

$ cat install_any_previous_r_package.sh
#! /bin/sh

# some R packages do not compile due to compile errors, try to repeat the procedure with diferent compiler and options set in ~/.R/Makevars , like these below:

##CXX = g++ -std=gnu++14
##CXX = g++ -std=c++11
##CC = gcc
#CXX = clang-10
#CXXFLAGS = -std=gnu++14
#CC = clang-10


# Usage:
# 
# cat package_listing.txt | sh install_any_previous_r_package.sh
#
# OR
#
# cat - | sh install_any_previous_r_package.sh
# pkgname1
# pkgname2
# ctrl+d

# comments for this installation script below:

# https://cran.r-project.org/mirrors.html
CRAN="https://mirrors.nic.cz/R/"
#CRAN="http://cran.r-project.org"
#

# parse one type of an error out of STDERR output of R when package fails to load after compilation
#grep ERROR installation.log.14081 | awk '{print $3}' | sed -e "s/‘//" -e "s/’//" -e 's/,//' | while read p; do

# TODO: handle multiple error lines from examples below
#ERROR: compilation failed for package ‘ranger’
#ERROR: compilation failed for package ‘Rhdf5lib’
#ERROR: configuration failed for package ‘hdf5r’
#ERROR: dependencies ‘edgeR’, ‘rjson’, ‘Rhdf5lib’, ‘beachmat’ are not available for package ‘scater’
#ERROR: dependencies ‘flowCore’, ‘flowUtils’ are not available for package ‘FlowSOM’
#ERROR: dependencies ‘mutoss’, ‘qqconf’ are not available for package ‘metap’
#ERROR: dependencies ‘scater’, ‘edgeR’, ‘beachmat’, ‘Rhdf5lib’ are not available for package ‘scran’
#ERROR: dependency ‘fastICA’ is not available for package ‘monocle’
#ERROR: dependency ‘flowCore’ is not available for package ‘flowUtils’
#ERROR: dependency ‘locfit’ is not available for package ‘edgeR’
#ERROR: dependency ‘multtest’ is not available for package ‘mutoss’
#ERROR: dependency ‘randomForest’ is not available for package ‘scmap’
#ERROR: dependency ‘ranger’ is not available for package ‘SCORPIUS’
#ERROR: dependency ‘Rhdf5lib’ is not available for package ‘beachmat’
#ERROR: lazy loading failed for package ‘clusterProfiler’
#ERROR: lazy loading failed for package ‘flowCore’
#ERROR: lazy loading failed for package ‘MAST’
#ERROR: lazy loading failed for package ‘plotly’
#ERROR: unable to collate and parse R files for package ‘Hmisc’

# provide a listing of package names, one line-by-line and the press ctrl+d
cat - | while read p; do
  if [ "$p" != "" ]; then
    mkdir -p $p;
    pushd $p;
    # fetch a list of source code release tarballs with their timestamps preserved
    wget --accept '*.tar.gz' --reject-regex '/\?C=[A-Z];O=[A-Z]$' --execute robots=off --recursive --level=0 --no-parent -nH -nd -nc "$CRAN"/src/contrib/Archive/"$p"/ ;
    if [ "$( ls -l $p*.tar.gz | wc -l )" -lt 1 ]; then
      # https://cran.r-project.org/src/contrib/lmds_0.1.0.tar.gz
      # https://cran.r-project.org/src/contrib/dyndimred_1.0.4.tar.gz

      # BUG: wildcards not supported in HTTP, so this does not work
      wget --accept '*.tar.gz' --reject-regex '/\?C=[A-Z];O=[A-Z]$' --execute robots=off --recursive --level=0 --no-parent -nH -nd -nc "$CRAN"/src/contrib/"$p"_*.tar.gz
    fi

    # list the release files from newest to oldest, skip '.' and '..' directory entries
    ls -lt --almost-all | awk '{print $9}' | while read rpackage; do
      if [ "$rpackage" != ""  -a  "$rpackage" != "."    -a   "$rpackage" != ".." ]; then
	# the below just handles one type of the error, see above for improvements
        R CMD INSTALL "$rpackage" 2>&1 | awk '{print $3}' | sed -e "s/‘//" -e "s/’//" -e 's/,//' >> packages_still_to_install.txt;
        RESULT=$?;
        if [ $RESULT -eq 0 ]; then
          echo "Installed $rpackage, breaking out of the while loop"
          break
        else
          echo "$rpackage failed to install, trying next"
        fi
      fi
    done
    popd;
  fi;
done

All the tools you discovered are definitely more advanced but install.packages() and devtools themselves are already as well. They just need to be more friendly and while have already parsed the versions of packages and knowledge of their origin repos, they could easily try in a row older and older version. But I already wrote that. As you can see, I een did not get to impleement the recursive function which would parse-out of child R output listing of the additional dependencies. I hope this will be useful as a demo example.

Yeah, I want to avoid Docker as it is insecure and superseded by Singularity/Apptainers and for the virtual machines LXC are even a step further. I admit the renv and similar are needed if I was to use R more ofter. It is already nightmare to keep tracks of various virtualenv, conda, Virtualbox, VMware images, Docker, Singularity/Apptainer. And it is all overkill just to get blindly installed any previous package version to get further with a daily job.

I will study the tools you referenced in more depth next week (as long as somebody pays me for that). I was supposed to bring to a life a Docker image from about 2020 with R, shinyR and a Firefox as it main interface.