DataFrame distinct with multiple column does not work

[jayMcoder]

I am trying to use distinct with multiple column select on DataFrame. But I am not getting distinct rows in the result. Distinct with single column select on DataFrame works fine. Example
df.distinct(row => [row.columnA, row.columnB]).toArray()
even tried this
df.distinct(row => ({ columnA: row.columnA, columnB: row.columnB }).toArray()
Both returns same number of rows as original dataframe, was expecting distinct rows from the original dataframe

Single column select
df.distinct(row => row.columnA).toArray() works perfectly fine

I know I am missing something here on the multi-column selector. Could someone help. Thanks in advance.

[ashleydavis]

Thanks for your feedback.

Can you please submit a code example so I can look into the problem?

The example code should be easily runnable and have a package file. Please put it in a GitHub repo and give a link to the repo.

Please make sure I can install dependencies using "npm install" and run the code using "npm start".

Thanks for your help!

[ashleydavis]

Hi, is this still an issue or can I close it?

[jayMcoder]

@ashleydavis I did not get time to create a sample code. I will create sample data and code tomorrow.
Thanks.

[jayMcoder]

I am creating test data

[ashleydavis]

How did you go creating that test data?

[ashleydavis]

Closing this issue due to no response.

[rat-matheson]

Just a note: I was able to handle the multiple columns scenario by turning my columns into a string key.

for example:

df.distinct(row => [row.columnA, row.columnB].join('_').toArray()

I haven't looked at the implementation but I'm guessing that distinct is not doing a deep member equals but instead, is just comparing objs. The original posters approach would have resulted in different array objects with the same members and perhaps since the object instances were different, it failed to do the user's intended distinct. If this is the case for why it happened, just updating the documentation and adding an example would probably be enough to prevent other users from having the same issue.

[ashleydavis]

@rat-matheson thanks for the info.

Would you be interested in adding an example and updating the docs? I'd love to have you as a contributor.

[rat-matheson]

Yes, I can do that. For all the hard work you've put into making this library, it is the least I can do.

By the way, been using it for about a week and it is amazing. Great job. It is so nice being able to work in typescript and not having to refresh myself in R or python. I suspect it is probably not as fast as similar implementations in those languages but the gains in productivity are certainly a fair trade for me!

Here's the pull request: #106

[ashleydavis]

Thanks so much for your help and your kind words.

I'm starting to think about version 2 of DF now, so more good stuff coming ;)