Run some new data through the system

[metasoarous]

@krdav has some data we can run through the system in order to kick the tires

[metasoarous]

Update here is that after looking at the data with @krdav, it seems that he doesn't have the partition.csv files which cft presently requires. In theory, we could make it possible to only supply the partition-cluster-annotation.csv file, since this is where most of the data we need comes from. We get the partition step logprob from the partition.csv file but that could probably be lived without. I think the bigger issue at this point is that in talks with @psathyrella it seems there's an overall move towards a framing of things where a single partition-cluster-annotion.csv may not be as meaningful by itself, as it could represent data for multiple partition steps, making it more necessary to have a partition-cluster-annotation.csv file to tie everything together. So I'm a little worried that this relaxation could lead to confusion down the road. @matsen @psathyrella @krdav Thoughts on this?

In any case, for the moment, without work relaxing assumptions, I can't run on @krdav's data until he has more time to do a rerun. In the mean time, I was able to kick the tires in a sense by manually curating a small dataset for a focused run of data for Laura, and the process prompted me to make some positive steps towards making things a little easier to set up and use, as seen in some of the issues I've tackled on this milestone.

[psathyrella]

I don't think I'm understanding some context (for one thing, I always spit out the partition file and the cluster annotation file, so unless one of them was manually deleted you'd always have both of them), but in any case, yes, for laura's data we decided to mash the cluster annotations for all partitions into the one cluster annotation file, partly because that made it easier for cft, partly because it simpified the partis workflow. But that only kicks in if you use the non-default clustering options that we use for Laura's data (--write-additional-cluster-annotations). Unless you set that option, the partition specified expicitly in the partition file, and the partition implied by the cluster annotations in the cluster annotation file, are the same.

[krdav]

It's a collection of data files I made a while ago. I didn't use the partition.csv so it's not included. That's the best I have now I am afraid.

…

On 9 Jan 2018 18:14, "Duncan Ralph" ***@***.***> wrote: I don't think I'm understanding some context (for one thing, I always spit out the partition file and the cluster annotation file, so unless one of them was manually deleted you'd always have both of them), but in any case, yes, for laura's data we decided to mash the cluster annotations for all partitions into the one cluster annotation file, partly because that made it easier for cft, partly because it simpified the partis workflow. But that only kicks in if you use the non-default clustering options that we use for Laura's data (--write-additional-cluster-annotations). Unless you set that option, the partition specified expicitly in the partition file, and the partition implied by the cluster annotations in the cluster annotation file, are the same. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#235 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQO7g378WQUPR_VUcOoXETRNL5gzhW_Yks5tJBzzgaJpZM4RSS5c> .

[metasoarous]

Marking this as done now that we've gotten the jason-mg data processed.