Sorting vectors and dataframes with nils

[gnilrets]

I was starting to work on figuring out a solution for sorting vectors and data frames with nil values. I noticed you've got a pending test - https://github.com/v0dro/daru/blob/master/spec/vector_spec.rb#L488.

I'd argue that nils should be placed at the beginning of the vector.

1. SQL sorts nulls at the beginning.
2. Empty ('') and blank (' ') strings are functionally similar to nil when dealing with datasets, and they sort before all other strings. It would be strange to have empty/blank strings at the top and nils at the bottom.
3. nil data is usually "special" and if it's at the top of a sort, you're more likely to notice.

[v0dro]

Hmmm that makes sense. I'm good with placing nils at the top.

Additionally, sorting currently uses a handmade Ruby variant of quick sort, which makes it slower than say Array#sort. If you have a solution to doing this faster (maybe use Array#sort somewhere so things happen at C level and are fast) while keeping the functionality same, I'm all ears!

[gnilrets]

Why do you use a handmade variant of quicksort rather than Ruby's native sort?

[v0dro]

The problem was that I couldn't figure out of a way to change the index along with the data, hence I went ahead with a hand made quick sort so that control could be maintained over the index objects as well.

Now that I think of it, I think zipping the indexes and data into tuples would do the trick, but I'm not sure if this can be extended to DataFrame.

[lokeshh]

I think sort_by could be used to replace handmade variant. Isn't it?

[v0dro]

Maybe only for Daru::Vector. DataFrame requires sorting only those specific columns that too in a nested manner.

[lokeshh]

Now that I think of it, I think zipping the indexes and data into tuples would do the trick, but I'm not sure if this can be extended to DataFrame.

I went ahead and took the liberty to implement your idea. There's a speed up.

DataFrame requires sorting only those specific columns that too in a nested manner.

I am not sure what that means.

[lokeshh]

DataFrame requires sorting only those specific columns that too in a nested manner.

Oh I get what it means now. I am thinking of sorting the DataFrame multiple times depending upon number of columns given. Say the columns to sort by are [:a, :b]. So, first we sort by :b and then by :a and we'll have the sorted DataFrame. Would that be fast?

[v0dro]

Yea but when you sort by :a first, only the elements in :b that correspond to equal elements in :a should be sorted, the sort cascades onto successive vectors, and this order can change depending on the use case.

[v0dro]

Solved in #67 and #63.