Using renv for Reproducible R Environments

Who this is for: anyone working in a lab project, which all use renv. Before you start: Getting Started with EpiModelHIV walks you through your first project setup; this page explains what renv is doing and how it works on the HPC.

Different projects need different versions of the same R packages (for example, two branches of EpiModelHIV-p). The renv package gives each project its own package library and records the exact versions in a lockfile, so that you, your collaborators, and the HPC all run the same code.

The pieces

• renv.lock: every package and version the project uses on your computer (models plus analysis packages).
• renv.lock.hpc: a smaller lockfile with only what the simulations need on the HPC (EpiModelHIV, EpiModelHPC, ARTnetData, and their dependencies). make_em_workflow() in R/hpc_configs.R regenerates it from your local library each time you build a workflow, so commit and push it before every HPC run.
• The project library (renv/library/): where the packages are installed. Not committed to git.
• The cache: renv keeps one copy of each package version per user (on the HPC in ~/.cache/R/renv) and links it into each project, so a second project with the same versions installs in seconds.

What we commit to git

In our projects (all created from the EpiModelHIV-Template) git tracks .Rprofile, renv.lock, and renv.lock.hpc. The whole renv/ folder and .Renviron are ignored. Our .Rprofile is lab code: it activates renv when renv/activate.R exists, and otherwise prints a message telling you how to set up the project. Do not add .Rprofile to .gitignore. (renv's own default is to also commit renv/activate.R and renv/settings.json; we regenerate them instead.)

On your computer

Install renv once with install.packages("renv"). Then, in a new project created from the Template, follow R/00-setup.R:

1. Run renv::init() and choose 1: Restore the project from the lockfile. (In a non-interactive session renv::init() does not install anything; use renv::init(bare = TRUE) followed by renv::restore() instead.)
2. Restart R.
3. Install packages with renv::install() (for example renv::install("EpiModel/EpiModelHIV-p@<branch>")), record them with renv::snapshot(), and commit the updated renv.lock.

If renv::init() fails partway (for example on a download error), fix the cause, restart R, and run renv::init() again with option 1. Never run renv::restore() by itself while renv/activate.R is missing: the project is not active yet, so renv installs into your personal R library instead of the project's.

After you git pull changes from a collaborator, restart R. The Template's .Rprofile runs renv::status() at startup; if it reports differences, run renv::restore().

EpiModelHIV-p and ARTnetData are private repositories, so renv needs your GitHub personal access token to install them. Put it in the project's .Renviron (see Getting Started with EpiModelHIV); a project .Renviron hides your user ~/.Renviron, so a token stored only there is not seen inside the project.

On the HPC

You usually do not run renv by hand on the HPC. The first step of every workflow (step_tmpl_renv_restore(), added by make_em_workflow()) runs git pull and then renv::restore(lockfile = "renv.lock.hpc") on a compute node. See Running Project Workflows on the HPC.

One-time setup on the HPC:

1. Clone the project over SSH into your project folder: cd /projects/epimodel/<netid>/ then git clone git@github.com:EpiModel/<Project>.git. Do not put a token in the clone URL; git would store it in plain text in .git/config.
2. Put your GITHUB_PAT=<your token> line in ~/.Renviron on the HPC and make it private with chmod 600 ~/.Renviron (see Using Git on the HPC). Project .Renviron files are not committed, so on the HPC R reads your user ~/.Renviron.

To restore by hand (for example before an interactive session), use a batch job run from the project root:

#!/bin/bash
#SBATCH --job-name=renv-restore
#SBATCH --partition=epimodel
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --output=renv-restore_%j.out

cd /projects/epimodel/<netid>/<Project>
. /projects/epimodel/spack/share/spack/setup-env.sh
spack unload -a
spack load r@4.5.1   # the R version in your project's R/hpc_configs.R
Rscript -e "renv::init(bare = TRUE, load = FALSE)"
Rscript -e "renv::restore(lockfile = 'renv.lock.hpc')"

A first restore of the Template's renv.lock.hpc (about 80 packages built from source) takes about 5 minutes on 4 CPUs; later restores reuse the cache and take seconds.

Two rules:

• Start R from the project root. R reads .Rprofile from the folder it starts in, and that is what activates the project library. From any other folder you get your personal library instead.
• Restore on a compute node, never on the login node. The login node runs Rocky Linux 8.9 and the compute nodes 8.10, and renv keeps a separate library for each (renv/library/linux-rocky-8.9/ and renv/library/linux-rocky-8.10/). Packages installed on the login node are not the ones your jobs load.

Messages you may see

• A startup message telling you to run renv::init(bare = TRUE) or R/00-setup.R: the renv/ folder does not exist yet. This is normal right after cloning.
• renv 1.3.1 was loaded from project library, but this project is configured to use renv 1.2.3 (or similar version numbers): your renv is newer than the one recorded in renv.lock. Harmless.
• GitHub authentication credentials are not available, then error downloading 'https://api.github.com/repos/EpiModel/EpiModelHIV-p/tarball/...' with [error code 22] or [error code 56], followed by Rolled back installation: R cannot see a valid GITHUB_PAT. The token is missing, expired, lacks access to the private repositories, or (on your computer) is in ~/.Renviron but not in the project .Renviron. Fix the token, restart R, and run the restore again; the packages already built are in the cache.
• An HTTP 401 error when installing from GitHub: the token has expired or been revoked. Make a new one.

More detail is on the renv website.