assertDataFrameEquals() Failure due to Different Row Order

[toddleo]

Hi,

I'm not sure if this issue is raised earlier after some searches.

It's common to have the same DataFrames with different row order. Hence my test case fails sometimes, and occasionally it successes. Is there a better way to compare DataFrames?

[spova]

Hi,
Shouldn't you sort them before comparison? Like this:

assertDataFrameEquals(expected.orderBy(col("ID").asc), output.orderBy(col("ID").asc))

[toddleo]

Hi @spova ,

Imagine I have more than one column in my DataFrame. Let's say 5 (ID, name, gender, etc.). Indeed you can sort by ID, however, the rest will stay unsorted, which makes them un-comparable. You may explicitly sort every column at the same time but it is non-trivial and not elegant coding work to do.

[noleto]

Hi folks,
I'm experiencing the same problem when comparing DataFrame with different number of partitions, i.e. row order isn't' strictly the same but all row are present. I took a look at @smadarasmi fix and it seems a good behaviour for testing purposes (sort by all columns before comparing dataframes).

@smadarasmi any plans to PR your fix into the main repo?
Thanks,

[zbstof]

Hello. Is this going to be implemented?
This should be built-in functionality, imho, akin to containsInAnyOrder from Java's Hamcrest library

[smadarasmi]

@noleto I don't have write access to this repo.

[nsutcliffe]

+1 for this

[nsutcliffe]

@smadarasmi you'll need to resolve the build errors on your PR #228

[smadarasmi]

@nsutcliffe The build fails on this step: Initializing download: http://www-us.apache.org/dist/spark/spark-2.2.2/spark-2.2.2-bin-hadoop2.7.tgz

It returns 404, not found.

Don't think it is related to my change.

[holdenk]

Oh yeah that's a good point, Spark changed it's release packaging so the older versions are available for download from the normal mirrors anymore and I haven't had a chance to update the travis file to point to the new version yet.

[nsutcliffe]

@smadarasmi in the mean time you could check the build is fine by updating .travis.yml, find the link to spark-2.2.2 (line 25) and replace with:
http://archive.apache.org/dist/spark/spark-2.2.2/spark-2.2.2-bin-hadoop2.7.tgz
@holdenk would it be ok to merge like that?

[holdenk]

If you wouldn't mind updating the travis file in your PR it would just be able to run CI and merge as normal? Otherwise I can do a quick PR to do that.

[nsutcliffe]

Do you want it on 2.2.3 or 2.2.2?

[holdenk]

Lets do 2.2.3 it should remain on the mirrors for longer.

[pablogomez93]

Hi :D! There are any news about this proposal? I think it's a huge feature, it adds a lot of value to the library for a very tiny changes.

[smadarasmi]

@holdenk Does it look okay for merging?

[minnieshi]

can this also work for spark 2.2?
currently i could not really compare 2 data frame equality as they have different order....