Converting Clustering Partitions from "Tab-Separated Labels per Line" Format to .mci Format for RCL

[mbertagna]

I'm working with externally generated clustering partitions, where each partition is stored in a file with tab-separated labels on each line (identical to the output format of mcl with the --abc flag). My goal is to use these partitions as input for RCL, but I need to convert them to the .mci format, which is the accepted format for RCL.

I initially tried using the mcxload -etc command for this conversion. While it successfully converted the files to .mci format, the RCL output was problematic— it produced the same clustering result for every resolution level. Upon investigation, I discovered that the .mci format generated by mcxload -etc differs from the format created by the rcl mcl command for clustering.

How can I effectively convert my externally generated clustering partitions to the .mci format that RCL requires?

[micans]

For the mcxload step make sure to have a tab file ready (that maps labels to consecutive indexes), make sure that each input files uses only those labels and all those labels and use that tab file as follows with mcxload. My tab file mydata.tab looks like this:

0  nul
1  een
2  twee
3  drie
4  vier
5  vijf
6  zes
7  zeven
8  acht
9  negen
10 tien
11 elf

My mycls1.label clustering file:

nul   een   vier  vijf  zes   negen
twee  drie  zeven acht  tien  elf

Then

mcxload -etc-ai mycls1.label -strict-tabr mydata.tab -o mycls1.mci
mcxload -etc-ai mycls2.label -strict-tabr mydata.tab -o mycls2.mci
...

You also need the network in mci format; do you already have that? For now I assume you do. I'm calling it graph.mci here. I've done the above cluster loading for a small set of examples locally; then when I do grep dim mycls*.mci graph.mci I get

mycls1.mci:dimensions 12x2
mycls2.mci:dimensions 12x3
mycls3.mci:dimensions 12x7
graph.mci:dimensions 12x12

indicating that they share the same row domain (which corresponds to the labels; the 2 3 and 7 indicate numbers of clusters).
If those clusterings are in a directory data, and you also have the network (graph.mci) and tab (mydata.tab) there, then

rcl setup YOURTAG -n data/graph.mci -t data/mydata.tab data/mycls*

should get the setup going, and I hope this works. Let me know of course if there are issues or if you don't have the network yet in the right format. Apologies for the not overly user-friendly experience.

[mbertagna]

Thank you @micans for this thorough explanation! I did already have my network in .mci format, so this was exactly what I needed.

[micans]

Cool, let me know if there are issues - I'm also mildly interested to know if there are no issues and the output is usable; both technically and whether the resulting RCL clustering makes sense in relationship to the input. Although RCL does little more than stack the inputs along (to my mind) the path of least resistance. Anyway, that's up to you.

Edit: I'm pretty sure you have this sorted, but for future reference and other people's benefit, it is important that all the clusterings (mycls1.mci etc above) and the network file (graph.mci) all used/correspond to the exact same tab file (mydata.tab).

[mbertagna]

I absolutely will. I encountered a minor issue with the existing RCL implementation, but I was able to resolve it by making a small change, and I've submitted a pull request for it. The RCL output clusters appear to make sense in relationship to the output based on my analysis so far. I opted to use RCL because I'm clustering genes based on co-expression, and I needed a consensus clustering method that could reconcile different flat clusterings with varying levels of granularity. So far, so good!

[micans]

Sounds all good! thanks for the fix, now merged.

[micans]

I'd like to add a caveat for a particular case where this RCL implementation is probably not giving helpful advice, although it does provide the information needed. If you have only few (ballpark five or less) input clusterings for a large-ish data set (say 10k or more; for expression data I assume your data set is somewhere in the range 5k-40k genes), then the RCL values are potentially not very differentiated. It will always construct a tree simply by taking the sorted RCL values and applying single link clustering. If there are ties it will just arbitrarily take one of the tying values. This could lead to a situation (if there are many ties) where two clusters of non-trivial size are created and then merged, all at the same linkage value. However, this situation will be visible from the joinval column in the rcl.sy.*.txt output. It can also be informative to look at a histogram of the val column in the rcl.join-order output.
How many clusterings do you have?
I'll just mention that it should be relatively quick to generate a set of about 12 clusterings with mcl, which could give something to compare to. mcl has not been used a lot for expression data, but in the Enright lab, with Tom Freeman (at that time in Edinburgh), we got good results with it (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2041979/ -- see also http://micans.org/mcl/man/clmprotocols.html#array). I'll add that it is not my intention to flog mcl. My experience with RCL is with both mcl and Leiden clusterings, and I'm curious how other cluster method fare when combined with RCL; the point here is the small caveat regarding the number of clusterings, and the option to compare to mcl (or another method).

[micans]

[I realised most of my 'answers' should have been 'replies'].
Ah, you're using mcl and Leiden, cool. I have looked (for the preprint kidney data set) into combining/using 12 mcl + 12 Leiden clusterings as a 24-clustering input ensemble, but it made the paper and the research too entangled. It would be better as a separate research strand, so I focused on them separately. They are also a bit different in nature and probably a bit more internally consistent when used separately (as the preprint qc indicates), so I think initially keeping them apart is the way to go.

As a question (combining) it's always hovering there of course. Some speculation: I don't think there is a such a thing as a single correct (hierarchical) clustering -- different methods give different views. As such, maybe combining mcl and Leiden could lead to a slightly out-of-focus view by trying to reconcile different angles. On the other hand, it's easy to argue that where different methods agree it should point to strong cluster structure. Against that, the trade-off may be that this strong cluster structure is paid for by a fragmented view in large parts of the data set.

What would be really interesting as the next step is to use the script rcl-qc to produce heat maps and granularity plots as in the preprint. I think this could genuinely give some insight into both clustering ensembles, and how well the range of inflation values / resolution values work for each (and how to change/add to them, potentially). rcl-qc -h gives some information, such as R packages required (ggplot2 viridis). Touch wood this works / is feasible (the script is quite hairy, requires R), if you give it a try. With those R packages installed, rcl-qc qc TAG (same TAG as used by rcl) should work.

[micans]

@mbertagna did you pursue this?

[mbertagna]

Hi @micans,
I apologize for the delayed response. I pursued this and was able to use the rcl-qc command to generate heat maps and granularity plots for MCL clusterings I generated with the rcl mcl command. However, when I try to run rcl-qc with the Leiden clusterings I converted from an external format to .mci, rcl-qc does not work. Instead, the following is printed to stdout: No match on line [d=6592 d1=4642 d2=1950 nn=20326 c1=43 c2=61 n1=TAG/out.leiden_clustering_1.mci n2=TAG/out.leiden_clustering_2.mci ]. Let me know if there is a fix for this issue. Thank you!

[micans]

Hello, nice to hear rcl-qc works - ish. The issue is that rcl-qc parses the file names and expects to find a string of the form r\d\d+ or I\d\d+ in each file name (an "r" followed by two or more digits or an "I" followed by two or more digits). This is to encode resolution, respectively inflation values. So if you change, for example, out.leiden_clustering_2.mci to out.leiden_clustering_r20.mci (encoding resolution 2) and out.leiden_clustering_2.5.mci to out.leiden_clustering_r25.mci then it should work.

[mbertagna]

@micans Ah that makes sense and worked! Thank you very much!