Building a matrix and filtering

Building a matrix and k-mers filtering

We will consider that a file samples.txt exists, which stores a set of samples (one per line). Every sample should be represented at least by one input file. E.g.:

killifishretinaA kfA_1.fastq.gz kfA_2.fastq.gz
killifishretinaB kfB.fastq.gz
killifishretinaC kfC_1.fastq.gz kfC_2.fastq.gz
killifishretinaD kfD_1.fastq.gz kfD_2.fastq.gz

Building a matrix for later utilization

To build a matrix for later utilization (in the form of binary .kmcdb and .stats files), the user may run the following command:

./mkmc --reuse-db -k 27 -- samples.txt results tmp

The generated files .kmcdb and .stats may be an input of MKMC for various postprocessing scenarios (normalization, correlation, differential k-mers analysis, cross-validation, entropy, dimensionality reduction; see Matrix postprocessing). --reuse-db both keeps the binary files and tries to use them (if they exist).

Building a matrix in the user-readable form

To build a matrix in the text form, the user may run the following command:

./mkmc -k 27 -o matrix -- samples.txt results tmp

MKMC will generate results_matrix file with the whole matrix of both k-mers sequences and per-sample counts. Just a set of k-mers in as FASTA file may be generated:

./mkmc -k 27 -o fa -- samples.txt results tmp

MKMC will generate results.fa file with k-mers sequences present in the matrix.

Also --reuse-db flag may be used to get a matrix in the form of binary files for later utilization (as in Building a matrix for later utilization). E.g.:

./mkmc --reuse-db -k 27 -o matrix -- samples.txt results tmp

Note that by default MKMC counts and then processes k-mers in the canonical form only. To process original k-mers use -b flag.

Building a matrix for non-FASTQ input files

By default MKMC expects FASTQ files as samples input data. To change the format, the user may use -f option, e.g.:

./mkmc -k 27 -f fa -o matrix -- samples_fasta.txt results tmp

for exemplary samples stored in FASTA files:

killifishretinaA kfA.fasta.gz
killifishretinaB kfB.fasta.gz
killifishretinaC kfC.fasta.gz
killifishretinaD kfD.fasta.gz

Another supported -f option values are -f fq for FASTQ (default) and -f mf for multi-FASTA.

Mixing formats is not supported. Compression method (gzip or none) is chosen automatically.

Building a matrix along with counts postprocessing

The matrix may be built simultaneously with its postprocessing (see Computing correlation for further information about computing correlation and correlation_phenotype.txt example). In such a situation --reuse-db (see Building a matrix for later utilization) parameter is not necessary (but useful to perform another postprocessing later). For example, to compute Kendall Tau correlation on counts normalized with frequency count method, the user may run the following command:

./mkmc -k 27 -n freq --corr kendall -p correlation_phenotype.txt -- samples.txt results tmp

Postprocessing may be performed in various ways. For all the possibilities see Matrix postprocessing and for running multiple postprocessings in a single run see Bulk processing.

Filtering k-mers along with building the matrix

To exclude k-mers appearing in the input files too rarely (e.g. suggesting they are an effect of sequencing errors), the user may specify minimal (and maximal) counts of k-mers that will be taken into account. The ones appearing less (more) than this value will be omitted or in the matrix will be treated as having counts equal to 0. For example, to exclude k-mers appearing at least twice, the user may run the following command:

./mkmc --reuse-db -k 27 --ci 2 -- samples.txt results tmp

To exclude k-mers appearing more than $10^9$ times, the user may run the following command:

./mkmc --reuse-db -k 27 --cx 1000000000 -- samples.txt results tmp

Note that the above options work independently for every sample and in the next stages the filtered k-mers from the sample will be treated as non-existent.

To exclude k-mers represented in too low number of samples, the user may specify two parameters: --thr-rat to define a fraction of samples which have to contain k-mer with count no lower than the specified threshold, and --thr to give the threshold. It is convenient e.g. to keep k-mers from genes expressed in specified fraction of samples. To exclude k-mers that appear less than twice in more than half of samples, the user may run the following command:

./mkmc --reuse-db -k 27 --thr 2 --thr-rat 0.5 -- samples.txt results tmp

To exclude k-mers which are not present in all the samples (and, finally, obtain a matrix with positive counts only), the user may run the following command (this is very strict filtering, probably with limited application scope):

./mkmc --reuse-db -k 27 --thr 1 --thr-rat 1.0 -- samples.txt results tmp

Note that --thr parameter refers to k-mers possibly previously filtered by k-mer counting (--ci and --cx parameters). To include a specified set of k-mers only (if present in input data and adequate number of times, obviously), the user may prepare a file with those k-mers in FASTA format (one k-mer per entry or e.g. an entire chromosome/contig per entry) or raw sequences, one per line, then run the following command:

./mkmc --reuse-db -k 27 --flt kmers.fasta -- samples.txt results tmp

Note that by default MKMC counts and then processes k-mers in the canonical form only. Thus, the user does not need to pass k-mers, e.g. both AAAAA and TTTTT, unless uses flag -b, which turns converting k-mers to the canonical form off.