Distance matrix from classification amd class2tree

[morgan-sparks]

Session Info

Just a quick question, not really a bug. I used the classfication() function for a list of species (includes insects, plants, molluscs, etc.) I have to download taxonomic hierarchy and then converted it to a tree using class2tree(). Works great! Obviously these are not true phylogenetic relationships, but I was curious how distances are determined for the phylo object. I am guessing it has to do with the taxonomic hierarchy and that a genus is so far from a family, a family so far from an order and so on, but there are no distinctions made for distance between plants and reptiles and reptiles from amphibians. I am trying to make variance-covariance matrix to control for phylogenetic non-independence in a statistical model and I am wondering how far off my mark I am using this method. Thanks!

[sckott]

thx for your question @morgan-sparks - @trvinh can you give some details?

It'd be worth adding a more detailed description to the function documentation too as it's pretty thin on what is being done within the function.

[trvinh]

Hi @morgan-sparks ,
yes, you were right, the classification is basically based on the taxonomy hierarchy string. Given a list of species, first the function will collect all possible taxonomy ranks and their corresponding IDs for each species. Then, it will try to align the rank vectors, just like doing a sequence alignment (with each position is a taxonomy rank, instead of a nucleotide or an amino acid), and replace the ranks by their taxonomy IDs. After that, this aligned ID matrix will be used to cluster species that have similar taxonomy string together (with hierarchical clustering function hclust).

For example, this is the rank vectors for 3 species:

SpecA: strain, species, family, phylum, kingdom, superkingdom
SpecB: strain, species, genus, family, kingdom, superkingdom
SpecC: strain, species, phylum, kingdom, superkingdom

Then our aligned rank matrix will look like:

Name	strain	species	genus	family	phylum	kingdom	superkingdom
SpecA	strain	species	NA	family	phylum	kingdom	superkingdom
SpecB	strain	species	genus	family	NA	kingdom	superkingdom
SpecC	strain	species	NA	NA	phylum	kingdom	superkingdom

It will be converted into this ID matrix (note: any missing ranks will have a pseudo ID from the previous rank):

Name	strain	species	genus	family	phylum	kingdom	superkingdom
SpecA	11	21	21	41	51	61	71
SpecB	12	22	31	41	61	61	71
SpecC	13	23	23	23	52	61	71

That aligned ID matrix will be used for clustering. As you can see, our tree will be:

((SpecA, SpecB), SpecC),

since SpecA and SpecB belong together to the family 41, and all three share the same kingdom 61.

So, the classification clusters the taxa not only based on the ranks (e.g. species, family, or phylum,...), but also on the actual taxonomy clade (specified by the IDs). Which means, it can cluster Arabidopsis within plant clade and snakes into reptiles. The more info you have in the aligned ID matrix (taxa with detailed taxonomy string; and enough taxa that cover all possible ranks), the higher resolution you can get for your taxonomy tree. For example, this is the first some lines of a real aligned ID matrix I am working with (not exactly the same as the one class2tree delivers, I just want give you an example how much data this matrix can/should have):

abbrName	ncbiID	fullName	strain	norank_491	norank_57678	norank_56615	norank_1086053	isolate	forma	varietas	formaspecialis	subspecies	species	norank_600669	norank_44542	speciessubgroup	norank_2642122	norank_2619919	norank_2627626	norank_1919231	norank_38063	norank_2622230	norank_2642239	norank_2629106	norank_2618147	norank_2625095	norank_2641301	norank_2621923	norank_256003	norank_2627676	norank_2621901	norank_2644672	norank_1104572	norank_2642439	norank_1234666	norank_2619537	norank_2646925	norank_2643926	norank_2635519	norank_2638681	norank_2624677	norank_2685009	norank_2644212	norank_2631227	norank_2144190	norank_2622732	norank_69483	norank_2643768	norank_2641396	norank_2668073	norank_2648758	norank_1655640	norank_2639520	norank_2634974	norank_2620651	norank_2641160	norank_2634096	norank_2648975	norank_2684913	norank_1700837	norank_132567	norank_2626337	norank_2634179	norank_2629569	norank_2623058	norank_1727652	norank_2684908	norank_2636082	norank_886737	norank_2638413	norank_2638438	norank_2632030	norank_2630403	norank_1679063	norank_2648658	norank_2642017	norank_2631942	norank_2630810	norank_2625419	norank_2565780	norank_2629969	norank_2648942	norank_2648404	norank_2631116	norank_2624956	norank_2610901	norank_2499238	norank_2752537	norank_857194	norank_220671	norank_254878	norank_2641196	norank_2633939	norank_2678336	norank_2640012	speciesgroup	norank_44537	subgenus	genus	norank_1535325	subtribe	norank_1304792	norank_651142	norank_1593277	norank_2116545	norank_114584	norank_2683658	norank_218105	norank_1113537	norank_1142503	norank_588816	norank_1803510	family	norank_469895	norank_715341	tribe	norank_588815	norank_1648033	subfamily	norank_359160	norank_147370	norank_2601530	norank_337687	norank_1912919	superfamily	norank_2601529	norank_74971	norank_104431	norank_43741	norank_37567	norank_43738	parvorder	norank_480117	norank_480118	infraorder	norank_33351	norank_33349	norank_33347	suborder	norank_33343	order	norank_159987	norank_33083	norank_355688	norank_9263	norank_91836	norank_314147	norank_6970	norank_4734	superorder	norank_71275	norank_186626	norank_1437201	norank_1489908	norank_1437010	subcohort	norank_1489872	norank_91827	norank_9347	norank_123369	norank_123368	norank_123367	norank_123366	norank_123365	cohort	norank_186625	norank_1489341	infraclass	norank_32525	norank_71240	norank_4447	subclass	norank_1437183	norank_1520881	norank_85512	class	norank_2283796	norank_2290931	norank_2692248	norank_2283794	norank_2683659	norank_2683660	norank_715962	norank_715989	norank_404260	norank_436492	norank_58024	norank_436491	norank_436489	norank_436486	norank_8492	norank_1329799	norank_32561	norank_8457	norank_32524	norank_78536	norank_58023	norank_716546	norank_3193	norank_32523	norank_1338369	superclass	norank_117571	norank_117570	norank_7776	norank_7742	subphylum	norank_197562	norank_197563	norank_716545	phylum	norank_1783276	norank_112252	norank_2611352	norank_2696291	norank_554915	norank_33630	norank_1206795	norank_33634	norank_2697495	norank_2698737	norank_3208	norank_1935183	norank_1783275	norank_2611341	norank_1798711	norank_68336	norank_1783257	norank_33511	norank_88770	norank_1783272	norank_1783270	norank_1206794	norank_33317	subkingdom	norank_33213	norank_6072	kingdom	norank_999999005	norank_999999001	norank_999999002	norank_999999003	norank_999999004	norank_33154	superkingdom	norank_131567	root
ncbi100226	100226	Streptomyces coelicolor A3(2)	100226	100226	100226	100226	100226	100226	100226	100226	100226	100226	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1902	1477431	1477431	1477431	1883	1883	1883	1883	1883	1883	1883	1883	1883	1883	1883	1883	1883	1883	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	2062	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	85011	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	1760	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	201174	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	1783272	2	131567	1
ncbi10090	10090	Mus musculus	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	10090	862507	10088	10088	10088	10088	10088	10088	10088	10088	10088	10088	10088	10088	10088	10088	10066	10066	10066	10066	10066	10066	39107	39107	39107	39107	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	1963758	1963758	9989	9989	9989	9989	9989	9989	314147	314147	314147	314146	314146	314146	314146	314146	1437010	1437010	1437010	1437010	9347	9347	9347	9347	9347	9347	9347	9347	9347	9347	32525	32525	32525	32525	32525	32525	32525	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	32524	32524	32524	32524	32524	32523	1338369	8287	117571	117570	7776	7742	89593	89593	89593	89593	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	33511	33511	33511	33511	33511	33511	33511	33213	6072	33208	33208	999999001	999999002	999999003	999999004	33154	2759	131567	1
ncbi10116	10116	Rattus norvegicus	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10116	10114	10114	10114	10114	10114	10114	10114	10114	10114	10114	10114	10114	10114	10114	10066	10066	10066	10066	10066	10066	39107	39107	39107	39107	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	337687	1963758	1963758	9989	9989	9989	9989	9989	9989	314147	314147	314147	314146	314146	314146	314146	314146	1437010	1437010	1437010	1437010	9347	9347	9347	9347	9347	9347	9347	9347	9347	9347	32525	32525	32525	32525	32525	32525	32525	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	40674	32524	32524	32524	32524	32524	32523	1338369	8287	117571	117570	7776	7742	89593	89593	89593	89593	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	7711	33511	33511	33511	33511	33511	33511	33511	33213	6072	33208	33208	999999001	999999002	999999003	999999004	33154	2759	131567	1

Have I answered your question, Morgan? I hope that it can help! :-)
Best,
Vinh

[sckott]

thanks @trvinh - can you add some of that to the function documentation?

[trvinh]

thanks @trvinh - can you add some of that to the function documentation?

@sckott : yes, I can. But where should I add it to? Directly to the Rd block for each subfunction? But how can the users see that, since those functions are only internal?

[sckott]

I was thinking in the documentation for the class2tree function https://github.com/ropensci/taxize/blob/master/R/class2tree.R#L1-L52 - perhaps in the @details section

you can also add docs to specific internal methods if you want, and then use @keywords internal so that the manual file isn't exported (Seen in the listing of docs pages), but the user can still get to it by doing ?fxn-name

[trvinh]

I will try :-)

[sckott]

thank you!

[morgan-sparks]

Thanks to you both @sckott @trvinh, this was very helpful and a nice option for those of us trying to make basic distance based phylogenies that span very broad species lists.