Overview

Caution

SCNIC is in the process of a major version change and release (v1.0.0)! Please pay close attention to the version of SCNIC that you have installed and reference the appropriate documentation!!

SCNIC v1.0.0+ Overview

This page provides additional information on the types of analysis SCNIC can do and their optional/required inputs, associated output files, and command line usage examples. If you would like a more hands-on example, reference the tutorial here.

Correlation calculation

Correlations can be calculated: 1) within all features in a singular input table or 2) between all features in two separate input tables.

Important

Input tables don't necessarily have to be traditional OTU tables - they can also be tables with KO counts (from PICRUSt2) or metabolite concentrations (from targeted LC/GC-MS)! As long as the input table follows the expected format, with samples as columns and the wanted feature as rows, SCNIC can be used to calculate correlations between said features.

Option 1: within

The within method takes as input biom formatted files and forms co-occurrence networks using a user specified correlation metric.

usage: scnic within [-h] -i INPUT_LOC [-o OUTPUT_LOC] [-m {sparcc,spearman,pearson,kendall}] [-a {fdr_bh,fdr_by,fdr_tsbh,fdr_tsbky,bonferroni,holm}] [-s MIN_SAMPLE] [--procs PROCS]
                    [--sparcc_filter] [--sparcc_p SPARCC_P] [-v]

options:
  -h, --help            show this help message and exit
  -i INPUT_LOC, --input_loc INPUT_LOC
                        location of input BIOM file (default: None)
  -o OUTPUT_LOC, --output_loc OUTPUT_LOC
                        location and desired name of output directory (default: scnic_within_out)
  -m {sparcc,spearman,pearson,kendall}, --correl_method {sparcc,spearman,pearson,kendall}
                        correlation method (default: sparcc)
  -a {fdr_bh,fdr_by,fdr_tsbh,fdr_tsbky,bonferroni,holm}, --p_adjust {fdr_bh,fdr_by,fdr_tsbh,fdr_tsbky,bonferroni,holm}
                        p-value adjustment, default is Benjamini-Hochberg FDR (default: fdr_bh)
  -s MIN_SAMPLE, --min_sample MIN_SAMPLE
                        minimum number of samples present in (default: None)
  --procs PROCS         number of processors to use (default: 1)
  --sparcc_filter       filter as described in SparCC paper (default: False)
  --sparcc_p SPARCC_P   calculate p-value for sparCC R value, give number of bootstraps (default: None)
  -v, --verbose         give verbose messages to STDOUT (default: False)

Input(s)

Input	Type	Required	Additional Information
Feature counts table (-i/--input_loc)	path to biom format file	Y	Features can be OTUS, ASVs, or a specific taxonomic level
Desired output file location (-o/--output_loc)	file path	N	If not provided, will be placed in the current working directory under scnic_within_out
Correlation method (-m/--correl_method)	parameter	N	Default method is sparcc but pearson, spearman, and kendall are also supported
P-value adjustment method (-a/--p_adjust)	parameter	N	Default method is fdr_bh but fdr_by, fdr_tsbh, fdr_tsbky, bonferroni, and holm are also supported
Minimum number of samples a feature is present in (-s/--min_sample)	parameter	N	This can help filter out sparse features prior to correlation calculations
Number of processors (--procs)	parameter	N	Default is 1 but can be increased if desired
SparCC filter (--sparcc_filter)	parameter	N	If included, will filter features as described in the SparCC paper
Calculate p-value for SparCC correlations (--sparcc_p)	parameter	N	If included, will calculate p-value for SparCC r-value, given number of bootstraps
Verbose messages (--verbose)	parameter	N	If included, will give verbose messages to STDOUT

Outputs

Output	File Name	Additional Information
Feature correlations	within_{correlMethod}_correls.txt	Gives r-values for each feature pair in the input biom table. If pearson, spearman, or kendall correlations were calculated, raw and adjusted p-values are also given
Correlation network	within_{correlMethod}_correlation_network.gml	Gives the unfiltered feature correlation network (a filtered correlation network is output with module creation below)
Analysis log	SCNIC_within_{correlMethod}_log.txt	Gives detailed information on the analysis run (start/end time, duration, input/output file paths, number of samples/observations, correlation method, analysis type, and processors used)

Example usage

scnic within --input_loc /location/of/your/feature_table.biom \
             --output_loc /location/of/within_output_dir/ \
             --correl_method sparcc \
             --p_adjust fdr_bh \
             --procs 4 \
             --verbose

Option 2: between

The between method takes two biom tables as input and calculates all pairwise correlations between the tables using a selection of correlation metrics. A gml correlation network is output as well as a file containing statistics and p-values of all correlations.

Important

Unlike the within method above, the between method only supports spearman and pearson correlation calculations.

This means that the input feature tables used for between must be rarefied or normalized prior to calculating correlations!

usage: scnic between [-h] -1 TABLE1 -2 TABLE2 [-o OUTPUT_LOC] [-m {spearman,pearson}] [-a {fdr_bh,fdr_by,fdr_tsbh,fdr_tsbky,bonferroni,holm}] [-s MIN_SAMPLE] [--max_p MAX_P]
                     [--min_r MIN_R] [--sparcc_filter] [--procs PROCS] [-f] [-v]

options:
  -h, --help            show this help message and exit
  -1 TABLE1, --table1 TABLE1
                        location of first BIOM table to be correlated (default: None)
  -2 TABLE2, --table2 TABLE2
                        location of second BIOM table to be correlated (default: None)
  -o OUTPUT_LOC, --output_loc OUTPUT_LOC
                        location and desired name of output directory (default: scnic_between_out)
  -m {spearman,pearson}, --correl_method {spearman,pearson}
                        correlation method (default: spearman)
  -a {fdr_bh,fdr_by,fdr_tsbh,fdr_tsbky,bonferroni,holm}, --p_adjust {fdr_bh,fdr_by,fdr_tsbh,fdr_tsbky,bonferroni,holm}
                        p-value adjustment, default is Benjamini-Hochberg FDR (default: fdr_bh)
  -s MIN_SAMPLE, --min_sample MIN_SAMPLE
                        minimum number of samples present in (default: None)
  --max_p MAX_P         max p-value to determine edges (default: None)
  --min_r MIN_R         minimum R to determine edges (default: None)
  --sparcc_filter       filter using parameters from SparCC publication (default: False)
  --procs PROCS         number of processors to use (default: 1)
  -f, --force           force overwrite output directory if it already exists (default: False)
  -v, --verbose         give verbose messages to STDOUT (default: False)

Input(s)

Input	Type	Required	Additional Information
First feature counts table (-1/--table1)	path to biom format file	Y	Features can be OTUS, ASVs, or a specific taxonomic level. These counts MUST be rarefied or normalized.
Second feature counts table (-2/--table2)	path to biom format file	Y	Features can be OTUS, ASVs, or a specific taxonomic level. These counts MUST be rarefied or normalized.
Desired output file location (-o/--output_loc)	file path	N	If not provided, will be placed in the current working directory under scnic_between_out
Correlation method (-m/--correl_method)	parameter	N	Default method is spearman but pearson is also supported
P-value adjustment method (-a/--p_adjust)	parameter	N	Default method is fdr_bh but fdr_by, fdr_tsbh, fdr_tsbky, bonferroni, and holm are also supported
Minimum number of samples a feature is present in (-s/--min_sample)	parameter	N	This can help filter out sparse features prior to correlation calculations
Maximum p-value to determine edges (--max_p)	parameter	N	WARNING: this parameter will most likely throw an error if used!! Any correlation network edges larger than this threshold will be excluded
Minimum r-value to determine edges (--min_r)	parameter	N	Any correlation network edges smaller than this threshold will be excluded
SparCC filter (--sparcc_filter)	parameter	N	If included, will filter features as described in the SparCC paper
Number of processors (--procs)	parameter	N	Default is 1 but can be increased if desired
Force overwrite of output directory (-f/--force)	parameter	N	If included, will force overwrite the output directory if it already exists
Verbose messages (--verbose)	parameter	N	If included, will give verbose messages to STDOUT

Outputs

Output	File Name	Additional Information
Feature correlations	between_{correlMethod}_correls.txt	Gives r-values, p-values, and adjusted p-values for each feature pair combination for the two input biom tables, (filtered by --min_r if provided)
Correlation network	between_{correlMethod}_crossnet.gml	Gives the feature correlation network (filtered by --min_r if provided)
Analysis log	SCNIC_between_{correlMethod}_log.txt	Gives detailed information on the analysis run (start/end time, duration, input/output file paths, number of samples/observations, correlation method, analysis type, and processors used)

Example usage

scnic between --table1 /location/of/your/feature_table1.biom \
              --table2 /location/of/your/feature_table2.biom \
              --output_loc /location/of/between_output_dir/ \
              --correl_method spearman \
              --p_adjust fdr_bh \
              --procs 4 \
              --verbose

Module creation

Regardless of whether you chose to calculate co-occurrence networks within features in one table or between features in two separate tables, the resulting {within/between}_correls.txt file serves as an input for the module creation step. Using the correlation network generated previously, SCNIC finds modules of co-occurring observations by finding groups of observations which all have a minimum pairwise correlation value. Modules are summarized and a new biom table with observations contained in modules collapsed into single observations are returned. This biom table along with a list of modules and their contents are output. A gml file of the network that can be opened using network visualization tools such as cytoscape is created which contains all observation metadata provided in the input biom file as well as module information. Please be aware that the networks output by this analysis will only include positive correlations as only positive correlations are used in module finding and summarization.

usage: scnic modules [-h] -i INPUT_LOC [-o OUTPUT_LOC] [--max_p MAX_P] [--min_r MIN_R] [--method {naive,k_cliques,louvain}] [-k K_SIZE] [-g GAMMA] [--table_loc TABLE_LOC]
                     [--prefix PREFIX] [-v]

options:
  -h, --help            show this help message and exit
  -i INPUT_LOC, --input_loc INPUT_LOC
                        location of correlation output from 'scnic within' or 'scnic between'. will be 'scnic_{within/between}_out' if default was used. (default: None)
  -o OUTPUT_LOC, --output_loc OUTPUT_LOC
                        location and desired name of the output directory (default: scnic_make_modules_out)
  --max_p MAX_P         maximum p-value to determine edges, p must have been calculated (default: None)
  --min_r MIN_R         minimum correlation value to determine edges (default: None)
  --method {naive,k_cliques,louvain}
                        method to be used for determining modules (default: naive)
  -k K_SIZE, --k_size K_SIZE
                        k value for use with the k-clique communities algorithm (default: 3)
  -g GAMMA, --gamma GAMMA
                        gamma value for use with louvain modularity maximization, between 0 and 1 where 0 makes small modules and 1 large modules (default: 0.1)
  --table_loc TABLE_LOC
                        biom table used to make network to be collapsed (default: None)
  --prefix PREFIX       prefix for module names in collapsed file (default: module)
  -v, --verbose         give verbose messages to STDOUT (default: False)

Input(s)

Input	Type	Required	Additional Information
Feature correlations (-i/--input_loc)	path to correlations .txt file	Y	Generated by running either within or between
Desired output file location (-o/--output_loc)	file path	N	If not provided, will be placed in the current working directory under scnic_make_modules_out
Maximum p-value to determine edges (--max_p)	parameter	N	Any correlation network edges larger than this threshold will be excluded. Input feature correlations MUST contain a p-value or this will fail!
Minimum r-value to determine edges (--min_r)	parameter	Y	Any correlation network edges smaller than this threshold will be excluded
Module creation method (--method)	parameter	N	Default method is naive (shared minimum distance algorithm) but louvain and k_cliques are also supported
K-value to use with k_cliques (-k/--k_size)	parameter	N	Default is 3, but can be changed if desired. This is only applicable if module method is k_cliques
Gamma value to use with louvain (-g/--gamma)	parameter	N	Default is 0.1, but can be changed to another value between 0 and 1 if desired (where 0 makes small modules and 1 large modules). This is only applicable if module method is louvain
Feature counts table (--table_loc)	path to biom format file	Y	Features can be OTUS, ASVs, or a specific taxonomic level and MUST be the same input used to calculate the feature correlations
Module prefix (--prefix)	parameter	N	Default is "modules" but can be changed if desired
Verbose messages (--verbose)	parameter	N	If included, will give verbose messages to STDOUT

Outputs

Output	File Name	Additional Information
Modules	modules.txt	Gives the modules found and the features that belong in them
Feature counts collapsed on modules	collapsed.biom	Returns the original feature counts table with additional rows summing the counts of all features within a module
Correlation network	module_correlation_network.gml	Gives the module correlation network (filtered by --min_r or --max_p)
Analysis log	SCNIC_module_log.txt	Gives detailed information on the analysis run (start/end time, duration, input/output file paths, number of samples/observations, correlation method, analysis type, and processors used)

Example usage

scnic modules --input_loc scnic_within_out/within_sparcc_correls.txt \
              --output_loc /location/of/modules_output_dir/ \
              --min_r 0.3 \
              --method naive \
              --table_loc /location/of/your/feature_table.biom \
              --verbose