Skip to content

A Hands On Introduction to Conda: October 6, 2021

Jump to bottom
Saranya Canchi edited this page Oct 7, 2021 · 1 revision

A Hands-On Introduction to Conda

When: Wednesday, Oct 6th, from 10:00 am-12:00 pm PDT

Instructors: Saranya Canchi

Moderators: Rayna Harris, Jeremy Walter

Description: This free 2-hour hands-on tutorial will introduce you to the package and environment manager, Conda. You'll learn how to set up conda environments and manage software installations. We'll discuss other applications, such as packaging software, running workflows, or creating binders all with conda!

While we wait to get started --

  1. ✔️ Have you looked at the pre-workshop resources page?

  2. 📝 Please fill out our pre-workshop survey if you have not already done so: Click here!

  3. 💻 Open a web browser - e.g., Firefox, Chrome, or Safari. Open the binder by clicking this button: Binder

Put up a ✋ on Zoom when you've clicked the launch binder button! (It takes a few minutes to load, so we're just getting it started before the hands-on activities)

Introductions

Have you heard of the NIH Common Fund Data Ecosystem?

Put up a ✔️ for yes and a ❎ for no!

We are at UC Davis and part of the training and engagement team for the NIH Common Fund Data Ecosystem, a project supported by the NIH to increase data reuse and cloud computing for biomedical research.

You can contact me at srcanchi@ucdavis.edu.

Goals for today:

  • learn about conda and how to use it
  • learn the basics of software installation, software dependencies, and isolated environments
  • learn about managing virtual environments

Why Conda?

  • OS-agnostic tool for package and environment management (meaning it works for Windows, Mac, and Linux!)
  • clean software installation along with dependencies
  • searches for compatible software versions
  • many packages available - python, R, etc.
  • user has admin permissions, good for use on HPC where you'd otherwise need to request software installations

What is an Environment?

Example Conda
School 🏫 Your computer 🖥️
Classrooms Conda environments
Room supplies 📝 Software packages
Students Input/output files 📁

Think of software environments like school classrooms, where the school is your computer. Schools have many classrooms and while they are independent rooms they are still connected by hallways. Students can walk into any room and each room will have its own set of supplies (i.e., desks, chairs, whiteboard, decorations, etc.).

For conda, the equivalent of a classroom and its supplies is a conda environment on your computer and its set of specifically downloaded software/tools. Input/output files, like students, can easily move between environments and just like a school can have multiple classrooms, conda can manage multiple conda environments. This is what makes it a great tool for maintaining different versions of the same software all on 1 computer!

Why do we need isolated software environments?

  • versioning: your research/project requires specific versions of software packages
  • reproducibility: you have a working analysis pipeline, but you want to experiment with new packages or features of existing packages without compromising your current workflow
  • repeatability: you want to replicate results from a publication
  • compatibility: your collaborators require you to use certain software/tools that are incompatibile with your current system

❓ Questions?

Let's get started!

If not done already:

  • 💻 Open a web browser - e.g., Firefox, Chrome, or Safari.
  • Open the binder by clicking this button: Binder

✔️ Did the binder load? Put up a ✋ on Zoom if you see Rstudio in your web browser.

Teaching tool notes:

  • binder vs. local computer
  • RStudio vs. terminal/Anaconda prompt

Set up Rstudio panels!

Initialize conda

⌨️ Copy/paste commands into the terminal OR run the commands from the workshop_commands.sh file in the binder.

Installer sets up two things: Conda and the root environment. The root environment contains the selected python version and some basic packages.

Image credit: Gergely Szerovay

Setup the conda installer and initialize the settings:

conda init

We will shorten command prompt to $:

echo "PS1='\w $ '" >> .bashrc

Re-start terminal for the changes to take effect (type exit and then open a new terminal).

:::success :heavy_check_mark: Put up a :hand: on Zoom when you've got a new terminal window and see this at the prompt: ~ $ :::

Conda channels: Searching for software

The channels are places that conda looks for packages. The default channels after conda installation is set to Anaconda Inc's channels (Conda's Developer).

In the absence of other channels, conda searches the default repository.

conda config --show channels

# get list of packages in base environment
# shows channel, package version, and build number for each package
# information for specifying packages (specific versions, builds, or default (latest)) 
conda list | less -S

Channels exist in a hierarchial order. By default:

Channel priority > package version > package build number

Image credit: Gergely Szerovay

See Resources for more info on channels!

conda-forge and bioconda are channels that contain community contributed software

  • Bioconda specializes in bioinformatics software (only supports 64-bit Linux and Mac OS, not Windows)
  • conda-forge contains many dependency packages
  • You can even install R packages with conda!

We will update the channel list order and add bioconda since we are using bioinformatic tools today. The order of the channels matters! The order tells conda where to start searching for software.

First, add the defaults channel:

conda config --add channels defaults
conda config --get channels

Then add the bioconda channel:

conda config --add channels bioconda
conda config --get channels

Lastly, add the conda-forge channel to move it to top of the list, following Bioconda's recommended channel order. This is because many packages on bioconda rely on dependencies that are available on conda-forge, so we want conda to search for those dependencies before trying to install any bioinformatics software.

conda config --add channels conda-forge
conda config --get channels

With this configuration, conda will start searching for packages in this order: 1) conda-forge, 2) bioconda, and 3) defaults

Another way to add channels is:

conda config --prepend channels bioconda

❓ Questions?

Install Software

We will install FastQC which is a software tool that provides a simple way to run quality control checks on raw sequencing data.

Search for software (fastqc):

conda search fastqc

Create conda environment and install fastqc:

conda create -y --name fqc fastqc

Activate environment:

conda activate fqc

Check fastqc version:

fastqc --version

✔️ Put up a ✋ on Zoom if your command prompt shows the name of the environment in parentheses: (fqc) ~ $

To go back to (base) ~ $ environment:

conda deactivate

High-throughput sequencing data quality control steps often involve fastqc and trimmomatic. Trimmomatic is useful for read trimming (i.e., adapters). There are multiple ways we could create a conda environment that contains both software programs:

Method 1: install software in existing environment

We could add trimmomatic to the fqc environment:

conda install -y trimmomatic=0.36
conda list # check installed software

We can specify the exact software version 👆 The default is to install the most current version, but sometimes your workflow may depend on a different version.

Method 2: install both software during environment creation

When you switch conda environments, conda changes the PATH (and other environment variables) so it searches for software packages in different places.

Let's check the PATH for method 1:

echo $PATH

You should see that the first element in the PATH changes each time you switch environments!

conda deactivate
conda create -y --name fqc_trim fastqc trimmomatic=0.36
conda activate fqc_trim 
conda list # check installed software
echo $PATH # path for method 2

The following methods use an external file to specify the packages to install.

Method 3: specify software to install with a YAML file

Often, it's easier to create environments and install software using a YAML file (YAML is a file format) that specifies all the software to be installed. For our example, we are using a file called test.yml. Let's start back in the (base) environment.

conda deactivate

The test.yml file contains the following in YAML format:

name: qc_yaml #this specifies environment name
channels:
    - conda-forge
    - bioconda
    - defaults
dependencies:
    - fastqc
    - trimmomatic=0.36

Create the environment - note the difference in conda syntax:

conda env create -f test.yml #since environment name specified in yml file, we do not need to use -n flag here
conda activate qc_yaml
conda list  # check installed software

See Resources for a 4th method of exporting exact environments.

Software can also be installed by specifying the channel with -c flag i.e.:

conda install -c conda-forge -c bioconda sourmash

Managing Environments

At this point, we have several conda environments! To see a list, there are 2 commands (they do the same thing!):

conda env list

or

conda info --envs

Generally, you want to avoid installing too many software packages in one environment. It takes longer for conda to resolve compatible software versions for an environment the more software you install.

For this reason, in practice, people often manage software for their workflows with multiple conda environments.

❓ Questions?

Running FastQC in the fqc Environment

If not already done, activate one of the environments we created, e.g.,:

conda activate fqc

Let's make sure the software was installed correctly:

fastqc --help

Output should look like:

FastQC - A high throughput sequence QC analysis tool

SYNOPSIS

        fastqc seqfile1 seqfile2 .. seqfileN

    fastqc [-o output dir] [--(no)extract] [-f fastq|bam|sam]
           [-c contaminant file] seqfile1 .. seqfileN
...

Download data (a yeast sequence file):

curl -L https://osf.io/5daup/download -o ERR458493.fastq.gz

Check out the data:

gunzip -c ERR458493.fastq.gz | wc -l

✔️ How many lines are in this fastq file? (add number to Zoom chat)

What does the fastq file look like?

gunzip -c ERR458493.fastq.gz | head

Run FastQC!

fastqc ERR458493.fastq.gz

✔️ Put up a ✋ on Zoom if you see an html file in the directory.

❓ Questions?

Exercise 1 ✍️

Hint! :bulb: You can use one of these approaches to install blast:

  • install the software in an existing env using conda install -y <name of the software>

  • create a new env using conda create -y --name <name of env> <software to install>

  • Second, try this example BLAST analysis in the conda environment. In this example, we are comparing the sequence similarity of a mouse protein sequence to a reference zebra fish sequence - as you might imagine, they are not that similar! But for today, this exercise will demonstrate running a quick analysis in a conda environment and bonus points if you find out how similar/dissimilar they are! (More details on BLAST and what each step is for here). Run each line of code below in the terminal:

    • Make a directory for the exercise files:
    mkdir exercise1
cd exercise1
    • Download with curl command and unzip data files:
    curl -o mouse.1.protein.faa.gz -L https://osf.io/v6j9x/download
curl -o zebrafish.1.protein.faa.gz -L https://osf.io/68mgf/download
gunzip *.faa.gz
    • Subset the data for a test run:
    head -n 11 mouse.1.protein.faa > mm-first.faa
    • Format zebra fish sequence as the blast database to search against:
    makeblastdb -in zebrafish.1.protein.faa -dbtype prot
    • Run a protein blast search with blastp!
    blastp -query mm-first.faa -db zebrafish.1.protein.faa -out myoutput.txt -outfmt 6

What does the output look like?

Note that if you conda deactivate, you can still access the input/intermediate/output files from the BLAST analysis. They are not 'stuck' inside the conda environment!

Exercise 2 ✍️

Conda allows you to revert to a previous version of your software using the --revision flag:

Usage:

# list all revisions
conda list --revisions

# revert to previous state
conda install --revision <number>

# for example:
conda install --revision 1

Earlier, we installed an older version of trimmomatic (0.36). Try updating it to the most recent version and then revert back to the old version.

Hint! :bulb: You can do this exercise in any of the conda environments we created earlier with trimmomatic. You can update software with conda update <software name>

Bonus activity

Mamba is a faster reimplementation of conda. At the moment, both are great tools and occasionally some things work better for one vs. the other.

You can replace conda with mamba for nearly all commands (conda activate still required to activate environments).

Try comparing the speed to run commands with mamba vs. conda using the time software, for example:

time mamba create -n mamba_test -y blast=2.9.0

time conda create -n conda_test -y blast=2.9.0

Output for mamba:

real    0m19.689s
user    0m16.232s
sys     0m2.471s

Output for conda:

real    0m50.241s
user    0m43.308s
sys     0m5.408s

Tidying up

To remove old conda environments:

conda env remove --name <conda env name>

To remove software:

conda remove <software name>

Be sure to save any work/notes you took in the binder to your computer. Any new files/changes are not available when the binder session is closed!

For example, select a file, click "More", click "Export":

What else can we use Conda for?

  • Reproducing analyses

    • Why: removes the guesswork on what version of software to use or how to install it!

    For example, it's a lot easier to replicate a software environment from a publication or share software versions used in your own analysis using conda (particularly method 3 & 4 mentioned above ways to create conda environments from YAML or exact package list files).

  • Packaging software

    • Why: to share your cool software with the world!

    The gist is that you write a recipe that has all the specs about your software that is submitted to a channel, e.g., conda-forge or bioconda. Once correctly formatted and tested (with continuous integration automation) so it works on different operating systems, it's added to the channel and the world can use and install your software with conda!

  • Running parts of analysis workflows in their own environment

    • Why: avoid software version conflicts, easier to keep track of software/versions used for a specific analysis, easier for others to reproduce

    This is a very helpful feature for workflows. For example, in Snakemake workflows, you can specify that each step (called a "rule") is executed in an isolated conda environment by adding a conda: directive:

    rule fastqc_raw:
    input: "rnaseq/raw_data/{sample}.fq.gz"
    output: "rnaseq/raw_data/fastqc/{sample}_fastqc.html"
    params:
         outdir="rnaseq/raw_data/fastqc"
    conda: "rnaseq-env.yml"
    shell:
          """
          fastqc {input} --outdir {params.outdir}
          """

  • Creating binders

    • Why: teaching tool for software or analysis demos, share reproducible analysis

    Examples:

Wrapping up

There are many actions you can perform with conda environments. Today we have mentioned these!

  • init
  • config
  • search
  • create
  • activate/deactivate
  • list
  • remove
  • update
  • revert
  • export

📝 Please fill out our post-workshop survey! We use the feedback to help improve our training materials and future workshops. Link here: https://forms.gle/kirsL9Y4qMeEmAa87

❓ Questions?

Resources

Clone this wiki locally