-
Notifications
You must be signed in to change notification settings - Fork 12
Tutorial
The tutorial below is intended to familiarize users with the strengths and weaknesses of running SCNIC on their data. Anyone following it is encouraged to contemplate the questions posed throughout the tutorial.
For those running this tutorial for Madi's lab meeting on September 1, 2026, please follow the installation instructions under Lozupone Lab Meeting Specific Instructions below!
Lozupone Lab Meeting Specific Instructions!!
Since the new updates to SCNIC have not been pushed to bioconda, you will be installing the current development version for use during lab meeting.
1. Clone the dev_MA_2026 branch of the SCNIC GitHub repo
git clone -b dev_MA_2026 --single-branch https://github.com/lozuponelab/SCNIC.gitNext, you'll need to install all of SCNIC's dependencies via the included dev_env.yml file. If you haven't already, move into the SCNIC directory you just cloned:
cd SCNIC/Then, you can create the SCNIC development conda environment using the dev_env.yml file.
Apple Silicon (M-chip) Users:
If your computer is an M-chip Mac, you'll need to set the environment subdirectory to osx-64 since some of SCNIC's dependencies do not support the computer architecture.
CONDA_SUBDIR=osx-64 conda env create -f dev_env.ymlLinux/WSL Users:
If you plan to run this tutorial on Linux, WSL, or an intel-based Mac, you don't have to worry about setting and environment subdirectory and can install it per usual.
conda env create -f dev_env.ymlNow that all of SCNIC's dependencies are installed into a new conda environment named dev-scnic, you use pip to install the development version of SCNIC.
Activate the conda environment you just created:
conda activate dev-scnicMake sure that you're still in the SCNIC directory you cloned and then run the following command:
pip install -e .Verify the installation:
scnic --helpCongratulations, you've officially installed the development version of SCNIC and are ready to proceed with the tutorial!
Caution
If you are running this tutorial for Madi's lab meeting on September 1, 2026, please following the installation instructions above and disregard those below this message.
If you have not installed SCNIC, do so now via conda, mamba, or pixi. Make sure that you are installing version 1.0.0 as older versions of SCNIC on bioconda will not be compatible with this tutorial!
Follow the appropriate installation instructions below based on your computer's operating system:
Linux, WSL, and intel Macs
## create a separate environment with only scnic and its dependencies installed
conda create -n scnic-v1.0.0 -c bioconda -c conda-forge scnic==1.0.0
## verify installation was successful
conda activate scnic-v1.0.0
scnic --helpApple Silicon Macs
Since some of SCNIC's dependencies do not support the apple silicon architecture, SCNIC will need to be installed with the Rosetta emulator.
## create a separate environment with only scnic and its dependencies installed
CONDA_SUBDIR=osx-64 conda create -n scnic-v1.0.0 -c bioconda -c conda-forge scnic==1.0.0
## verify installation was successful and make sure that the environment continues to use rosetta
conda activate scnic-v1.0.0 && conda config --env --set subdir osx-64
scnic --helpImportant
ATTENTION: Those running this tutorial for Madi's lab meeting DO NOT need to download the files below, they are already included in the SCNIC directory that you cloned under SCNIC/tutorial/mouse-cecal-genera.biom and SCNIC/tutorial/bile-acid.biom!!
The data used for this tutorial was taken from another Lozupone lab project that investigated the impacts of dietary fiber and fat on C. difficile infection and the gut microbiome after antibiotic challenge in a murine model. For additional information, see the associated publication and GitHub repository.
Tutorial data is uploaded to the SCNIC Github repository here. Specifically, the following two files will need to be downloaded:
SCNIC can be run on ASVs, OTUs, or a specific taxonomic level (bonus: feature tables composed of targeted/untargeted metabolomics concentrations can be used too but we'll talk more about that later). Running SCNIC on ASVs/OTUs would be the most taxonomically precise but typically their tables are more sparse (i.e. you have more zeros). For the purposes of this exercise, I've collapsed the OTU table to genus-level features so that the outputs will be more informative.
Since this tutorial uses compositional microbiome data, we'll run SCNIC within using sparcc as the correlation method to calculate pairwise comparisons between all genera. When you run the following command, you will most likely get a warning from SparCC saying that some OTUs only have one unique permutation. For now, say that you would like to continue since we'll be addressing this issue later in the tutorial.
Tip
If we wanted to use any of the other correlation methods (spearman, pearson, or kendall), our input table would need to be rarefied or normalized!
scnic within --input_loc tutorial/mouse-cecal-genera.biom \
--output_loc scnic-genera/within/ \
--correl_method sparcc \
--procs 2 \
--verboseNote
Question #1: Does the within_sparcc_correls.txt file have a p-value column?
Answer:
No! Why do you think that is?
Explanation: Calculating correlations using sparcc only returns the effect size (r-value) rather than correlation significance because the SparCC algorithm can't calculate p-values analytically. If desired, you can include the --sparcc_p flag and number of bootstraps (minimum of 1000) to generate bootstrap tables, compute pseudo-correlations, and extract the adjusted p-values. Be warned; the process of calculating p-values in this manner drastically increases compute time! Although, I would also argue that p-values may not be necessary for this context.
For the purposes of this exercise, let's see what happens when we run SCNIC within using spearman as the correlation method instead.
scnic within --input_loc tutorial/mouse-cecal-genera.biom \
--output_loc scnic-genera/within-spearman/ \
--correl_method spearman \
--p_adjust fdr_bh \
--procs 2 \
--verboseNow when we look at scnic-genera/within-spearman/within_spearman_correls.txt we have two new columns, one with the raw p-values and the other with the Benjamini and Hochberg FDR adjusted p-values. So we now know that the sparcc method doesn't return p-values but spearman does.
Note
Question #2: Based on what we found above, do you think p-values are important to module creation?
Answer:
Also, no!
Explanation: SCNIC doesn't actually support module creation based off of p-values and solely relies on effect size (r-value).
We chose to run SCNIC within on an input table where the ASVs had been collapsed to the genus-level which decreases sparsity of the data, even if we also lose some precision. However, we didn't check how sparse the data was even after collapsing to the genus-level and we got a warning from sparcc related to 6 OTUs that only have one unique permutation. To further decrease sparsity of the data, what if we required each genus to be found in at least 5 samples to be included in our correlation analysis?
Let's rerun SCNIC within using sparcc including the --min_sample flag and see if it throws any warnings this time.
scnic within --input_loc tutorial/mouse-cecal-genera.biom \
--output_loc scnic-genera/within-min5/ \
--correl_method sparcc \
--min_sample 5 \
--procs 2 \
--verboseSince no warnings were thrown from SparCC when we used our --min_sample filter, we should proceed with our analysis using those correlations for module creation.
Note
Question #3: Do you think applying the --min_sample filter will affect module creation?
Answer:
Yes, but to what degree will depend on the overall sparsity of your data.
Explanation: As a rule of thumb, if your correlation calculations throw a warning related to sparse features, you should re-run SCNIC within with varying --min_sample cutoffs and proceed with the one that fits your data the best (or at least alleviates the warnings).
Next, let's take our correlation outputs from running SCNIC within to generate modules. These modules are created using only positive correlation values and require a minimum r-value threshold (--min_r).
Tip
Using a higher r-value threshold (closer to 1) will produce more strongly connected modules but with fewer features while a lower value (closer to 0) will produce more loosely connected modules that contain more of the features.
In total, SCNIC modules does three separate things:
- Defines modules using the correlation information and produces a
modules.txtfile describing the results (i.e. which genus is in which module?) - Generates a collapsed feature table where the counts of all features that were in the same module are summed and appended to the end
- Produces a
.gmlfile displaying the network that can be opened in network visualization software such as cytoscape
Let's see what genera are in the modules when we use our scnic-genera/within/within_sparcc_correls.txt where we didn't apply any filters. We're also going to use the naive method for module creation which applies the shared minimum distance (SMD) algorithm with a base r-value threshold of 0.3.
scnic modules --input_loc scnic-genera/within/within_sparcc_correls.txt \
--output_loc scnic-genera/modules/ \
--min_r 0.3 \
--method naive \
--prefix genus_module \
--table_loc tutorial/mouse-cecal-genera.biom \
--verboseAfter this initial run of SCNIC modules, do you think you could go back and answer Question #2 above? It appears that p-values are not vital to module creation but specifying a minimum r-value threshold is! This was an intentional choice made to ensure that modules are created based off effect size rather than correlation significance.
Note
Question #4: What happens to the modules when you rerun SCNIC modules with a --min_r of 0.8? What about 0.1 or 0.5? Are the same genera included?
Answer:
0.8: 0 modules with 0 observations
0.1: 19 modules with 275 observations
0.5: 10 modules with 140 observations
Explanation: As mentioned in the tip above, using a higher r-value threshold (closer to 1) is more likely to identify highly phylogenetically related microbes with overlapping niches while a lower value (closer to 0) bins a broader community of loosely correlated microbes but runs the risk of grouping features that should not have been. When we sent the r-value to 0.1 for this dataset, we lost about 10 modules and 100 features compared to when we used an r-value of 0.3 (27 modules with 395 observations). Finding the best r-value for module creation differs per dataset and users should always test multiple, comparing ANCOMBC results with and without running SCNIC on the feature table.
Now let's run SCNIC modules using our filtered correlations (where each genera must be found in at least 5 samples to be included) so we can answer Question #3.
scnic modules --input_loc scnic-genera/within-min5/within_sparcc_correls.txt \
--output_loc scnic-genera/modules-min5/ \
--min_r 0.3 \
--method naive \
--prefix genus_module \
--table_loc tutorial/mouse-cecal-genera.biom \
--verboseNote
Question #5: Are the modules similar if you run SCNIC modules using the louvain or k_cliques methods instead of naive?
Answer:
Even with using the same r-value threshold of 0.3, the modules are not similar when using louvain/k_cliques versus naive.
Explanation: Louvain identified 15 modules with 215 observations, with each individual module containing more genera than any of the modules identified using naive (SMD). Louvain performed similarly to running naive at an r-value threshold of 0.1 and thus may have the same issues with losing individually significant features and binning features that should not be. K-Cliques identified 6 modules with 84 observations with a k-value of 3 which is far less than naive or louvain.
What if we wanted to compare the newly formed genus-level modules to another dataset taken from the same samples? Thankfully, the project that produced the OTU table we're using above also quantified cecal bile acids on the same samples. Before we can compare the genus-level modules to bile acid concentration, we'll need to use SCNIC to calculate correlations within and generate modules of bile acids.
Since bile acid data is not compositional, we'll have to use spearman to calculate correlations.
scnic within --input_loc tutorial/bile-acid.biom \
--output_loc scnic-bile-acid/within-spearman/ \
--correl_method spearman \
--procs 2 \
--verboseWe got the same warning from spearman as we did above with the genus-level observations. Do you think we should apply a --min_sample filter?
scnic within --input_loc tutorial/bile-acid.biom \
--output_loc scnic-bile-acid/within-spearman-min5/ \
--correl_method spearman \
--min_sample 5 \
--procs 2 \
--verboseWhen we recalculate correlations with spearman using the --min_sample filter, the warnings disappear and we get correlation networks with 9 nodes and 36 edges.
MADI ADD SOMETHING ABOUT HOW THE BILE ACIDS ARE HIGHLY CORRELATED WITH EACH OTHER AND HOW THAT MAY CHANGE OUR MIN_R THRESHOLD!!
Since we've alleviated the warnings, we can feel comfortable creating modules using our --min_sample filtered spearman correlations.
scnic modules --input_loc scnic-bile-acid/within-spearman-min5/within_spearman_correls.txt \
--output_loc scnic-bile-acid/modules-spearman/ \
--min_r 0.3 \
--method naive \
--prefix bile_acid_module \
--table_loc tutorial/bile-acid.biom \
--verboseNote
Question #6: What do you notice about the contents of the bile acid modules? Are the bile acids related to each other?
Answer:
Yes, the bile acids are related to each other!
Explanation: In particular, module 0 contains all bile acids conjugated with taurine (one of which is the conjugated form of the precursor for lithocholic acid, taurochenodeoxycholic acid). Module 1 contains two prominent secondary bile acids (deoxycholic acid and lithocholic acid) and the precursor for deoxycholic acid, cholic acid (a primary bile acid). Module 2 contains two bile acids that are specific to mice (alpha muricholic acid and beta muricholic acid).
Now we can finally calculate pairwise correlations between the genus-level and bile acid modules we generated using their output collapsed.biom tables. Since sparcc doesn't support the kind of correlation comparisons we'll be doing, we'll have to use spearman instead.
scnic between --table1 scnic-genera/modules-min5/collapsed.biom \
--table2 scnic-bile-acid/modules-spearman/collapsed.biom \
--output_loc scnic-bile-acid-genera/between/ \
--correl_method spearman \
--p_adjust fdr_bh \
--procs 2 \
--verboseNote
Question #7: Is there potential biological significance to the highly correlated genera and bile acids?
Answer:
Potentially! Although, this would be a starting place for further hypothesis generation and testing.
Explanation: Upon consulting the output between_spearman_correls.txt file, we can see one example where genus module 6 (contains genera within the Lachnospiraceae and Ruminococcaceae families) was negatively correlated with bile acid module 0 (contains all conjugated bile acids) but positively correlated with bile acid modules 1-2 (contains mostly secondary bile acids and C. difficile inhibitors).
We could continue on to attempt to create modules based on the between correlations, but what if we first run the same calculations again but instead of using the collapsed bile acid biom table, we used the raw bile acid concentrations? You may have noticed that not many bile acids were included on the panel and thus, observations may not need to be collapsed to reduce the number of pairwise comparisons.
scnic between --table1 scnic-genera/modules-min5/collapsed.biom \
--table2 tutorial/bile-acid.biom \
--output_loc scnic-bile-acid-genera/between-full/ \
--correl_method spearman \
--p_adjust fdr_bh \
--procs 2 \
--verboseNote
Question #8: When you compare the correlations calculated between the two collapsed biom tables versus the entire raw bile acid table, which results are easier to read through? What are the pros and cons of each method? What are potential pitfalls of doing the analysis like this?
Answer:
This question is mainly rhetorical and meant to entice the user to think deeply about what analysis is best for the dataset. I've included some of my thoughts in the explanation below.
Explanation: In a way, it was nice to compare the genus and bile acid modules to one another but since the bile acid input table has very few features (only 16), generating bile acid modules for comparison with the genus modules may have been unnecessary. Visually, its much easier to look through which individual bile acids correlated with genus modules. Its hard to draw any concrete conclusions from these results since several genera within a module have the potential to influence bile acid pools in different ways. The results may have been more meaningful if I went back and re-generated the genus modules with a higher r-value threshold (0.5-0.7) to capture genera that are more phylogenetically similar. I would also like to bring up the fact that this method of comparison was mainly for tutorial purposes and isn't the most efficient way to determine associations between bile acid concentrations and microbiome genera.
Let's move on to generating modules based on either of the between methods we employed above. We'll need to use louvain as the module creation method instead of naive since naive doesn't work in this context (I need to find a better way to explain this).
scnic modules --input_loc scnic-bile-acid-genera/between-full/between_spearman_correls.txt \
--output_loc scnic-bile-acid-genera/modules \
--min_r 0.3 \
--method louvain \
--verboseNote
Question #9: Do you think that calculating correlations between genus-level abundances and bile acid concentrations provided additional insight into the data?
Answer:
For this particular dataset, no.
Explanation: Since we're mostly interested in whether bile acids correlate with the microbes that can alter them in some way, creating modules that bin the genus and bile acid modules together reduces granularity and seems unnecessary given the few bile acids we started with in the first place (16). There may be applications for this method when comparing other data types with several more features like untargeted metabolomics or shotgun metagenomics gene counts.
SCNIC is another method of feature reduction that increases statistical power, increases the accuracy of FDR (false discovery rate) measurements, and decreases feature dependence. The collapsed.biom table output from SCNIC modules is intended to be used in downstream statistical analysis like ANCOMBC for differential abundance calculations or random forest models.