Matrix postprocessing

Counts normalizing and matrix postprocessing

We will consider that the file samples.txt is present, as in Building a matrix and filtering, and the binary matrix (.kmcdb and .stats files) are already generated, as shown there.

Normalizing counts

k-mers counts may be normalized with one of the algorithms: frequency count, quantile, or DESeq2. The normalized counts may be dumped to a file and/or utilized to perform some of another postprocessing tasks. To normalize with frequency count, the user may run the following command:

./mkmc --reuse-db -k 27 -n freq -- samples.txt results tmp

This command line is useful with another postprocessing. However, the user may also dump the counts in the form of text matrix with --save-n flag:

./mkmc --reuse-db -k 27 -n freq --save-n -- samples.txt results tmp

MKMC will generate results_matrix_norm file with the whole matrix of both k-mers sequences and per-sample normalized counts.

Similarly, to normalize with another method, the user may pass q or deseq values of -n option, respectively. Only one method of normalization may be chosen in a single run.

Computing correlation

MKMC may determine correlation of normalized counts of every k-mer with the specified phenotype/design. There are available Kendall Tau, Pearson, and Spearman methods. First, the user has to create a text file, e.g. correlation_phenotype.txt, with consecutive one per-line (integer or real) phenotype/design values. As all correlations require reads normalization, it is necessary to determine one or more of the normalization methods (see Normalizing counts). The user may compute Kendall Tau correlation of reads normalized with frequency count normalization with the following command:

./mkmc --reuse-db -k 27 -n freq --corr kendall -p correlation_phenotype.txt -- samples.txt results tmp

for an exemplary design/phenotype file correlation_phenotype.txt:

10.2
9.0
9.4
11.9

The file contains $(\text{no. of samples})$ numbers (integers or reals), corresponding to the consecutive samples in samples.txt.

MKMC will generate results_kendall_tau file of k-mers with correlation values. Moreover, the following files with top k-mers in terms of correlation value will be generated: results_kendall_tau_top of k-mers with correlation values, results_kendall_tau_top.fa of k-mers sequences, results_kendall_tau_top_matrix of the whole matrix of both k-mers sequences and per-sample unnormalized counts.

Number of the top k-mers (by default 10000) may be changed with --n-top parameter, e.g.:

./mkmc --reuse-db -k 27 -n freq --corr kendall -p correlation_phenotype.txt --n-top 500 -- samples.txt results tmp

To normalize with another method, the user may pass another value of -n option. To choose another correlation method, the user may pass pearson or spearman values of --cor option, respectively. It is possible to compute correlation with a few methods in a single run, e.g.:

./mkmc --reuse-db -k 27 -n freq --corr kendall pearson sperman -p correlation_phenotype.txt -- samples.txt results tmp

For every method the results will be stored in 4 separate files, as for Kendall Tau.

Generating matrix for cross-validation

Performing correlation along with exclusion of some of samples may be used for cross-validation (see Computing correlation for further information about correlation, setting number of the top k-mers, and correlation_phenotype.txt example). The user may run cross-validation with --cv flag. To specify the number of samples to be included in a test set for every fold may use --leave option, $(\text{no. of samples})$ has to be multiple of $(\text{leave})$. E.g. --leave 1 means performing LOOCV, --leave p means performing LpOCV. MKMC will perform $f=(\text{no. of samples}) / (\text{leave})$ correlation runnings on remaining samples counts, and for every one a top-correlated k-mer matrix will be returned. It is warranted that every sample will be included in test set, but the order is random. E.g.:

./mkmc --reuse-db -k 27 -n freq --corr kendall -p correlation_phenotype.txt --cv --leave 2 -- samples.txt results tmp

MKMC will generate a series of 3 files for every of $f=(\text{no. of samples}) / (\text{leave})$ folds, marked as 1_f, 2_f, ..., f_f. The files will contain k-mers with correlations (e.g. results_1_f_top); k-mers with counts (results_1_f_top_matrix); FASTA with k-mers (results_1_f_top.fa). Every of the files contain top k-mers in terms of correlation value.

Performing differential k-mers analysis

MKMC may perform differential k-mers expression (DE) analysis of (sometimes normalized) counts of every k-mer with the specified phenotype/design. There are available ANOVA, DIDS, Signal to Noise ratio, T-Test, Wilcoxon-rank sum (Mann-Whitney U test) methods. First, the user has to create a text file, e.g. de_phenotype.txt, with consecutive one per-line classes of the phenotype/design. The classes may be text labels (e.g. positive, negative; or male, female). The user may compute ANOVA statistic of counts normalized with frequency count normalization with the following command:

./mkmc --reuse-db -k 27 -n freq --diff anova -c de_phenotype.txt -- samples.txt results tmp

for an exemplary design/phenotype file de_phenotype.txt:

female
male
male
female

The file contains $(\text{no. of samples})$ classes (labels), corresponding to the consecutive samples in samples.txt.

MKMC will generate results_anova file of k-mers with both ANOVA results and p-values.

Using a method generating no p-values like DIDS:

./mkmc --reuse-db -k 27 -n freq --diff dids -c de_phenotype.txt -- samples.txt results tmp

will generate results_dids file of k-mers with DIDS results. Moreover, the following files of top k-mers in terms of DIDS result will be generated: results_dids_top of k-mers with DIDS results, results_dids_top.fa of k-mers sequences, results_dids_top_matrix with the whole matrix of both k-mers sequences and per-sample unnormalized counts.

Number of the top k-mers (by default 10000) may be changed with --n-top parameter, e.g.:

./mkmc --reuse-db -k 27 -n freq --diff dids -c de_phenotype.txt --n-top 500 -- samples.txt results tmp

To normalize with another method, the user may pass another value of -n option. To choose another analyse method, the user may pass other --diff option values as in the following table. Note that various methods have different requirements (some of them require normalized, some unnormalized counts; some allows only two classess, some two or more; some produces result, which is furtherly postprocessed, some does not).

DE method	Normalization required	Maximal number of classes	Result postprocessing	p-value in output (correction possible)	--diff parameter value
ANOVA	yes	unlimited	no	yes	anova
DIDS	yes	unlimited	by default $\sqrt{x}$, where $x$ is a raw output of DIDS; see below	no	dids
Signal to Noise ratio	yes	2	no	no	snr
T-Test	no	2	$\ln{(x+1)}$, where $x$ is a raw output of T-Test	yes	ttest
Wilcoxon-rank sum (Mann-Whitney U test)	yes	2	no	yes	wrs

If DIDS is computed, the user may pass a parameter --dids-mode to specify postprocessing method: --dids-mode sqrt (default) for resulting $\sqrt{x}$; --dids-mode quadratic for resulting $x^2$; --dids-mode tanh for resulting $1 + \tanh{(3x - 3)}$, where $x$ is a raw output of DIDS.

It is possible to compute correlation with a few methods in a single run, e.g. (providing that de_phenotype.txt contains two classes only):

./mkmc --reuse-db -k 27 -n freq --diff anova dids snr ttest wrs -c de_phenotype.txt -- samples.txt results tmp

For every method the results will be stored in separate 4 files, as for ANOVA and DIDS.

Correcting p-values

If MKMC postprocesses counts with a method computing p-values (ANOVA, T-Test, Wilcoxon-rank sum (Mann-Whitney U test); see Performing differential k-mers analysis for differential k-mers analysis explained), they may be corrected. For example, Bonferroni correction may be performed by passing --pval-corr b option with the following command:

./mkmc --reuse-db -k 27 -n freq --diff anova -c de_phenotype.txt --pval-corr b -- samples.txt results tmp

To normalize with another method, the user may pass another value of -n option (see Normalizing counts). To choose another correction method, the user may pass bh (Benjamini-Hochberg), by (Benjamini-Yekutieli), or hb (Holm-Bonferroni) to --pval-corr option, respectively. Only one method of correction may be chosen in a single run.

MKMC will generate results_anova_cor file of k-mers with ANOVA results with corrected p-values instead of results_anova. Moreover, the following files of statistically significant k-mers only will be generated: results_anova_cor_significant of k-mers with ANOVA results with corrected p-values, results_anova_cor_significant.fa of k-mers sequences, results_anova_cor_significant_matrix with the whole matrix of both k-mers sequences and per-sample unnormalized counts.

Statistical significance level (by default 0.05) may be changed with --max-corrected-pval parameter, e.g.:

./mkmc --reuse-db -k 27 -n freq --diff anova -c de_phenotype.txt --pval-corr b --max-corrected-pval 0.01 -- samples.txt results tmp

Determining k-mers counts entropy

MKMC may compute entropy of every k-mer counts across samples.

./mkmc --reuse-db -k 27 --entropy -- samples.txt results tmp

Before computing, the counts are incremented, e.g. for the given counts 10, 0, 15 entropy is counted for 11, 1, 16 values.

MKMC will generate results_entropy file of k-mers with their entropy values. Moreover, the following files of top k-mers in terms of entropy values will be generated: results_entropy_top of k-mers with DIDS results, results_entropy_top.fa of k-mers sequences, results_dids_entropy_matrix with the whole matrix of both k-mers sequences and per-sample unnormalized counts.

Dimensionality reduction

MKMC may reduce dimensionality of the data by transforming normalized counts from $(\text{number of k-mers})\times(\text{no. of samples})$ dimensions to (by default) $2\times(\text{no. of samples})$. There are two reduction algorithms: PCA (--pca flag) and UMAP (--umap). The following command generates reduced dimension files with both of the algorithms (using single method is also possible).

./mkmc --reuse-db -k 27 -n freq --pca --umap -- samples.txt results tmp

MKMC will generate results_pca and results_umap files of $2\times(\text{no. of samples})$ coordinates and for PCA results_pca_variance file of variance and variance ration for every $(\text{no. of samples})$ dimension.

To change the number of output dimensions use --dimensions option, e.g.:

./mkmc --reuse-db -k 27 -n freq --pca --umap --dimensions 3 -- samples.txt results tmp

Note that dimensionality reduction may require much time or memory.