Implement IndexedFrame.duplicated with distinct_indices + scatter

[wence-]

Description

To obtain the duplicate rows in a dataframe we previously performed a drop-duplicates with a carrier column of row indices and then set entries in a boolean column to False for those row indices that remained. Furthermore, we were performing an unnecessary merge after the drop-duplicates call to obtain the row indices.

Note that the carrier column provides exactly the information that is computed internally in libcudf by cudf::detail::get_distinct_indices (called as part of cudf::distinct). We therefore promote get_distinct_indices to a public function (as cudf::distinct_indices) and replace the (unnecessary) merge plus iloc-based setting of the result with a call to libcudf.copying.scatter.

This provides a reasonable speedup (around 1.5x) for duplicated() on Series, and significantly improves performance of duplicated() on DataFrames, especially when providing a subset argument. Previously we would pay the cost in drop-duplicates of moving all columns of the distinct rows to the output table, even though we only actually needed the carrier "indices" column. Now we just obtain those indices directly, duplicated() scales only with the number of "active" columns. In some simple benchmarking this is between two and five times faster for tables with 10% distinct rows depending on the number of passive additional columns.

• Closes [PERF/ENH]: DataFrame.duplicated does unnecessary inner merge #14486

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[wence-]

Note that one could also imagine a distinct_by_key(table_view values, table_view keys), but the choice in these functions (which is deliberate policy, #3303 (comment)) is to just allow specifying keys by subsetting a table. However, if one only needs a subset of the table as output to continue, this will move more data in the gather phase than necessary: there's no way with the current distinct(table_view values, std::vector<size_type> key_columns) to say "use these columns as keys, but only gather these other columns".

[wence-]

Ready for another look @bdice, thanks!

[karthikeyann]

LGTM 👍

[wence-]

/merge