segment_matrix improvements (#30)

[lomereiter]

Note: there are a few changes in the output JSON files (only one dataset made it into the commit since that's what I test on). This is because order of arrivals and departures arrays' elements is not fixed. For testing against the original results I run jq '.components[].arrivals |= sort_by(.upstream, .downstream) | '.components[].departures |= sort_by(.upstream, .downstream)' on each chunk.

[josiahseaman]

Is this ready for review?
I'd like to not that occupants is just a precompute to make Schematize logic a bit simpler. It could be removed if the same code was changed in Schematize to be smarter (which may also slow the browser). Arrivals and departures, however are necessary I believe. They're derived from the order of links listed in the bin. I don't know that it would be simple to remove that requirement.

If you care about JSON size, I have a branch where I changed all the lists to dictionaries or sets which is a radically smaller file size https://github.com/graph-genome/component_segmentation/tree/experimental_v6_sparse_matrix. I ran into issues with this format particularly on the Schematize side. Now I think it wouldn't be worth it with all the other JSON format changes. Something to keep in mind though. If we can precompute things to make the browser faster that's good. However, if the large file size makes the file load slow in the browser, that's counter-productive. I'll leave it to your good judgement.

[josiahseaman]

The Python code here is really clean. I can't find anything to gripe about at all. If I understand correctly, you're saying the order the link columns are listed in the JSON is not deterministic. Example of them switching:

Since you've now seen the deep insides, I'd appreciate it if you read Minutiae: Link Column Ordering. This describes the specification, rather than the current reality. Short version: link column should stack from the inside out on a component as they're traversed, there are likely cases where no consistent sort is possible across all individuals.

I see no reason not to merge this in. Really, well done.
.

[josiahseaman]

Question: since we have data checked into the repo, is it going to generate a diff every time the same command is run?

[lomereiter]

Question: since we have data checked into the repo, is it going to generate a diff every time the same command is run?

No, it won't. The order now corresponds to the traversal of the sorted dataframe, there are no random choices involved.