arg_to_tskit fails with more than one mutation per site

[petrelharp]

For instance, running the example with --add_mutations on the convert call (and without --fit_to_data) fails with:

_tskit.LibraryError: Duplicate site positions. (TSK_ERR_DUPLICATE_SITE_POSITION)

Note that in tskit, each site is represented once so that all mutations at a given genomic position refer to the same site.

Perhaps this is not urgent because this workflow (without --fit_to_data) isn't recommended? If there is any threads or arg-needle-lib workflow that produces sites with multiple mutations, then this would be nice to have?

Happy to consult if it's not obvious what to do here - I haven't looked at the code.

[ArniFreyrG]

This is due to an assumption in the arg-needle-lib conversion function that's being run under the hood here that all mutations have perfect monophyly and so can be represented by a single site in the tree sequence. There are some differences between the way the ARG-Needle ARG, the tskit TreeSequence and the Threads ThreadingInstructions represent mutations that make them difficult to pass between one another. It works ok when the --fit_to_data flag is specified, since it guarantees monophyly of derived carriers in the output ARG. I've opened an issue for this on the arg-needle-lib repo.

[petrelharp]

Ah yes, sorry, I should have opened this issue over at arg-needle-lib.

[ArniFreyrG]

That's fine, good to have here as well, the Threads docs are insufficiently clear on this limitation and there should be a warning or a check somewhere before going into a conversion function that might fail, I'll see if I can't find a quick fix for that

[petrelharp]

Oh right! There should be an easy fix - this is exactly why we have deduplicate_sites (see the python API for docs). Probably, you've just got to run this function on the table collections before you write them out. Note that if you've got more than one mutation per site, you may need to run tsk_table_collection_compute_mutation_parents also.

edit: oh, I see you are doing this in python so never mind the C links

[petrelharp]

It looks like just inserting tables.deduplicate_sites() right here would do it.

(Hm, and I was also worried about ambiguity of ordering if you had more than one mutation on the same site and the same branch but with unknown times, but I see that all mutations have the same derived state, so that wouldn't matter if it did happen.)