Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix handling of hash collisions in row_group_slots #1979

Merged
merged 3 commits into from Oct 10, 2019
Merged

Conversation

nalimilan
Copy link
Member

Original code was correct but JuliaData/DataTables.jl#79 incorrectly moved break outside of the if.

Fixes #1978.

Original code was correct but JuliaData/DataTables.jl#79 incorrectly moved `break` outside of the `if`.
@nalimilan nalimilan mentioned this pull request Oct 9, 2019
Copy link
Member

@bkamins bkamins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for a quick fix of this. I have checked the logic, and it seems OK so I approve 😄.

@mattBrzezinski - it would be great if you had a chance to test it.

@nalimilan If I understand it correctly df = DataFrame(A=1:2_000_000) test by @mattBrzezinski already exposes the collisions and I think this should be small enough to be added to our tests.

@mattBrzezinski
Copy link

Good morning.

I can verify this fixes the issue later today, and write some unit tests for it as well!

@nalimilan would you like me to push commits to this branch? Or just create a new one for the unit-tests?

@nalimilan
Copy link
Member Author

I've added a test: can you just check that it reproduces the problem on 32-bit without the patch? Thanks!

@bkamins
Copy link
Member

bkamins commented Oct 9, 2019

Now I have four Julia versions installed in parallel to be able to develop DataFrames.jl 😄.

The test correctly reproduces the error prior to the change in this PR:

julia> gd = groupby(df, :A)
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] getindex at .\array.jl:728 [inlined]
 [2] group_rows(::DataFrame, ::Bool, ::Bool, ::Bool)
 [3] #groupby#157(::Bool, ::Bool, ::typeof(groupby), ::DataFrame, ::Array{Symbol,1})
 [4] #groupby at .\none:0 [inlined]
 [5] #groupby#158 [inlined]
 [6] groupby(::DataFrame, ::Symbol)
 [7] top-level scope at REPL[6]:1

@nalimilan
Copy link
Member Author

I've added some more checks to make sure that even if no error happens for some reason, we don't mix two different values that hash equal in the same group.

@nalimilan nalimilan merged commit 57aec33 into master Oct 10, 2019
@nalimilan nalimilan deleted the nl/collision branch October 10, 2019 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

32-bit BoundsError
3 participants