Fix attribute corruption

[darabos]

In LynxKite 4.2.1 I've discovered an attribute corruption bug. I've seen all values or everything except the first value disappear from an attribute. It's happening somewhat randomly, but not on small data. In my repro I'm importing a CSV with 600,000 lines, using it as a graph, and out of the 5 attributes there is a good chance that one or more will be affected.

I've tracked it to PulledOverVertexAttribute. It gets a good Parquet file as the input and writes out a bad Arrow file (undefined almost everywhere). I've added debug stuff in PulledOverVertexAttribute and saw that it sees the input attribute already as mostly undefined.

I've added debug stuff in unordered_disk_io.go and here I saw this in permutation[:10]: [0 0 0 419538 1 419539 209908 209909 209910 2]. That is not a permutation! Why does it have 0 three times?

We create this array by sorting an array of [0, 1, 2, 3, ...] in sortedPermutation. But the sort call did deviate slightly from Sorty's docs. (https://pkg.go.dev/github.com/jfcg/sorty#Sort) I didn't check if r != s. Seemed unnecessary I guess? What's wrong with swapping an element with itself?

But adding that if fixes the issue! Sorty sorts with a parallel algorithm. So I guess it's some horrible fallout from multi-threading? 🤢

[xandrew-lynx]

Sorry I missed this earlier. Looks good of course, thanks for tracking it down!

All this - but even the API itself... - really makes me wonder how on earth sorty does its magic!