hash_grouper with single complex types seems incorrect

[waitinfuture]

        if self.num_columns() == 1 {
            return self
                .columns
                .first()
                .unwrap()
                .as_materialized_series()
                .make_groups();
        }

For complex types such as struct, as make_groups() makes groups based on hash rather than actual value, when hash collision happens, the result might be incorrect. The same happens with make_unique_idxs.
cc @jaychia Please check if the understanding is correct, thanks!

[everettVT]

Hi @waitinfuture , Do you have a full minimum reproducible example you can share? That will help us understand the failure mode better.

[everettVT]

That particular result appears correct to me.

Should return 1.

[waitinfuture]

@everettVT Why should return 1? {'a': 0, 'b': 42}, {'a': 1, 'b': 3995947126312393897} are different items

[waitinfuture]

@everettVT I tested pyspark and returns 2:

>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, LongType
>>>
>>> spark = SparkSession.builder.getOrCreate()
>>>
>>> schema = StructType([
...   StructField("x", StructType([
...       StructField("a", LongType()),
...       StructField("b", LongType()),
...   ]))
... ])
>>>
>>> df = spark.createDataFrame([
...   ({"a": 0, "b": 42},),
...   ({"a": 1, "b": 3995947126312393897},),
...   ({"a": 0, "b": 42},),
... ], schema=schema)
>>> result = df.select('x').distinct().collect()
>>> print(len(result))
2

[waitinfuture]

The distinct result of Daft:

>>> df.select('x').distinct().show()
Daft Query ID: sharp-heron-242648
╭────────────────────────────╮
│ x                          │
│ ---                        │
│ Struct[a: Int64, b: Int64] │
╞════════════════════════════╡
│ {a: 0,                     │
│ b: 42,                     │
│ }                          │
╰────────────────────────────╯

(Showing first 1 of 1 rows)
>>>

The distinct result of Spark:

>>> df.select('x').distinct().show(100,100)
+------------------------+
|                       x|
+------------------------+
|                 {0, 42}|
|{1, 3995947126312393897}|
+------------------------+

[everettVT]

I see, apologies, thanks will look into this.