SwissDict #634

eulerkochy · 2020-06-14T17:47:10Z

SwissDict

Part of GSoC'20 work.
Initial code and tests.
Tests run and works.
As of now, the benchmarks aren't favorable.
I'll post the benchmarks once I finish the work on #517

Tasks remaining

Refactor code, and improve benchmarks.
Add docstrings and deploy doc
Add file to README

oxinabox · 2020-06-16T11:09:32Z

src/swiss_dict.jl

+import Base: setindex!, sizehint!, empty!, isempty, length, copy, empty,
+             getindex, getkey, haskey, iterate, @propagate_inbounds, ValueIterator,
+             pop!, delete!, get, get!, isbitstype, in, hashindex, isbitsunion,
+             isiterable, dict_with_eltype, KeySet, Callable, _tablesz, filter!


Do we always do this per file?
I thought we just did this at the top level file

it's there in robin_dict.jl. Nice suggestion, let me get to it and try to get it out from there as well. I'll remove this from here as well.

oxinabox · 2020-06-16T11:11:55Z

src/swiss_dict.jl

+copy(d::SwissDict) = SwissDict(d)
+empty(d::SwissDict, ::Type{K}, ::Type{V}) where {K, V} = SwissDict{K, V}()
+
+const AnyDict = SwissDict{Any,Any}


this is never used and seems unuseful and confusing

oxinabox · 2020-06-16T11:16:25Z

src/swiss_dict.jl

+    return dest
+end
+
+empty(a::AbstractDict, ::Type{K}, ::Type{V}) where {K, V} = SwissDict{K, V}()


Why are we chanign the default definition of empty for all AbstractDicts?
That doesn't seem right

oxinabox · 2020-06-16T11:17:31Z

src/swiss_dict.jl

+
+#hashindex(key, sz) = (((hash(key)%Int) & (sz-1)) + 1)::Int
+
+##Simd utilities


Suggested change

##Simd utilities

##SIMD utilities

Also fun, i haven't touched these llvmcall for SIMD in ages.

oxinabox · 2020-06-16T11:22:01Z

src/swiss_dict.jl

+@inline function _prefetchr(p::Ptr)
+    p = reinterpret(UInt, p)
+    if UInt === UInt64
+        Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """
+    %ptr = inttoptr i64 %0 to i8*
+    call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)
+    ret void
+    """), Nothing, Tuple{UInt}, p)
+    else
+        Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """
+    %ptr = inttoptr i32 %0 to i8*
+    call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)
+    ret void
+    """), Nothing, Tuple{UInt}, p)
+    end
+end


Can we get rid of the if by interpolating the type string in?

Suggested change

@inline function _prefetchr(p::Ptr)

p = reinterpret(UInt, p)

if UInt === UInt64

Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """

%ptr = inttoptr i64 %0 to i8*

call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)

ret void

"""), Nothing, Tuple{UInt}, p)

else

Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """

%ptr = inttoptr i32 %0 to i8*

call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)

ret void

"""), Nothing, Tuple{UInt}, p)

end

end

@inline function _prefetchr(p::Ptr)

p = reinterpret(UInt, p)

int_t = Int === Int64 ? "i64" : "int_t"

Core.Intrinsics.llvmcall(

("declare void @llvm.prefetch(i8*, i32, i32, i32)",

"""

%ptr = inttoptr $int_t %0 to i8*

call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)

ret void

"""),

Nothing, Tuple{UInt}, p

)

end

oxinabox · 2020-06-16T11:22:23Z

src/swiss_dict.jl

+
+@inline function _prefetchw(p::Ptr)
+    p = reinterpret(UInt, p)
+    if UInt === UInt64


similar to above, can we interpolate the type string in?

oxinabox · 2020-06-16T11:30:45Z

src/swiss_dict.jl

+end
+
+@propagate_inbounds function _slotget(slots::Vector{_u8x16}, i::Int)
+    #@boundscheck 0 < i <= length(slots)*16 || throw(BoundsError(slots, 1 + (i-1)>>4)


why is this commented out?

oxinabox · 2020-06-16T11:35:36Z

test/runtests.jl

-         "priority_queue",
-         "fenwick",
+         "priority_queue", 
+         "fenwick", 


I think maybe you did not intend to add this white-space?

rfourquet · 2020-06-16T11:47:59Z

Could you post a link to a reference on what are "Swiss dictionaries" ? It's difficult to find relevant results on search engines with these terms, also with "swiss map" ;-)

eulerkochy · 2020-06-16T11:57:25Z

@rfourquet SwissDict as such doesn't exist. It is based on Abseil's flat_hash_map, which in turn uses Swiss Tables, develeped by Google engineers. Here's the cppcon talk by Matt Kulukundis which inspired my implementation.

rfourquet · 2020-06-16T12:02:06Z

@eulerkochy Thanks!

…res.jl into swissdict

vtjnash · 2020-07-01T21:29:28Z

src/swiss_dict.jl

+    p = reinterpret(UInt, p)
+    if UInt === UInt64
+        Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """
+    %ptr = inttoptr i64 %0 to i8*
+    call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)
+    ret void
+    """), Nothing, Tuple{UInt}, p)
+    else
+        Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """
+    %ptr = inttoptr i32 %0 to i8*
+    call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)
+    ret void
+    """), Nothing, Tuple{UInt}, p)
+    end


Suggested change

p = reinterpret(UInt, p)

if UInt === UInt64

Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """

%ptr = inttoptr i64 %0 to i8*

call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)

ret void

"""), Nothing, Tuple{UInt}, p)

else

Core.Intrinsics.llvmcall(("declare void @llvm.prefetch(i8*, i32, i32, i32)", """

%ptr = inttoptr i32 %0 to i8*

call void @llvm.prefetch(i8* %ptr, i32 0, i32 3, i32 1)

ret void

"""), Nothing, Tuple{UInt}, p)

end

ccall("llvm.prefetch", llvmcall, Cvoid, (Ref{Int8}, Int32, Int32, Int32), Ptr{Int8}(p), 0, 3, 1)

end

Part of SciML/ColPrac#1 See: https://discourse.julialang.org/t/colprac-contributors-guide-on-collaborative-practices-for-community-packages/41850

Co-authored-by: Lyndon White <oxinabox@ucc.asn.au>

eulerkochy · 2020-08-29T13:19:04Z

K=Int64, 1e6 elements	Dict	SwissDict	RobinDict
make	65.17 MB / 79.772 ms	22.66 MB / 70.462 ms	106.66 MB / 155.695 ms
pop!	134.000 ns	300.000 ns	325.00 ns
empty!	69.189 μs	39.550 μs	2.389 ms
find-success	60.000 ns	212.000 ns	179.000 ns
find-failure	230.000 ns	122.000 ns	142.000 ns

…res.jl into swissdict

oxinabox · 2020-08-30T12:08:14Z

src/swiss_dict.jl

+    h.nbfull += (iszero(index & 0x0f) & (so==0x00))
+    _slotset!(h.slots, tag, index)
+    if index < h.idxfloor
+        h.idxfloor = index


do we need to fix the tests to make sure to cover this?

I can't seem to find a test to cover this. It's quite hard to find a test case specifically for this. But, this serves the purpose of h.idxfloor, so it isn't wrong.

eulerkochy added 3 commits June 14, 2020 23:10

write initial code for SwissDict, add tests

4002001

Merge branch 'master' into swissdict

eff12d4

undo changes in bench_robin_dict.jl

5ab6c06

eulerkochy requested a review from oxinabox June 14, 2020 19:13

oxinabox reviewed Jun 16, 2020

View reviewed changes

eulerkochy added 4 commits June 16, 2020 17:32

clean code in swiss_dict.jl

16ad2b8

clean code in swiss_dict.jl

30d7c2b

Merge branch 'swissdict' of https://github.com/eulerkochy/DataStructu…

cc7e327

…res.jl into swissdict

fix runtests.jl

3971b9d

vtjnash reviewed Jul 1, 2020

View reviewed changes

eulerkochy and others added 17 commits August 14, 2020 17:54

shorten llvm.prefetch call

11b7406

Add tests to increase coverage

6442ad8

match the behavior to Base.Dict, modify tests

ea52db5

Update Project.toml

35485ed

Link to ColPrac

12ce458

Part of SciML/ColPrac#1 See: https://discourse.julialang.org/t/colprac-contributors-guide-on-collaborative-practices-for-community-packages/41850

remove redundant iterate method

d53806f

initial code for red-black tree

27fbdd7

Add Base.copy,Base.copymutable for Sorted containers

87fb04d

Bump patch version number when adding Base.copy() overloads

b86940b

Add test that copy/copymutable preserve type

c8f0af1

insert! working for RBTree

6886aa7

implement delete!, change ==, import Base.show

c7e9472

fix minor bugs in delete!, add tests

1ef2324

add more tests, to cover cases by default

3c5bc53

remove un-necessary code, add tests to improve coverage

193cbf5

fix tests for 0.7

ba272fe

handle cases related to insert

937ec3f

eulerkochy and others added 16 commits August 28, 2020 22:29

export RBTree and deploy docs

951da2a

Update docs/src/red_black_tree.md

62800ed

Co-authored-by: Lyndon White <oxinabox@ucc.asn.au>

Apply suggestions from code review

b7eb1d0

Co-authored-by: Lyndon White <oxinabox@ucc.asn.au>

rebase to master, and apply code review suggestions

7a7f50b

Fix tests

af5fe0e

final suggestions applied

2f866fe

bump version

0d6a826

Add RBTree to README.md

8faf29b

Fix typo in RBTree docstring

677c382

Fix the overloaded docs in v0.18.1

a7a836d

bump version

ee70afe

write initial code for SwissDict, add tests

033a58c

undo changes in bench_robin_dict.jl

90109c4

use prefetchr

1f5801a

Merge branch 'master' into swissdict

a322c74

Merge branch 'master' into swissdict

ef3dbae

eulerkochy added 4 commits August 29, 2020 18:51

add SwissDict to bench_dict, and improve benchmarks

6db72e7

Merge branch 'swissdict' of https://github.com/eulerkochy/DataStructu…

2bb69d8

…res.jl into swissdict

add docstrings

e2bc9a4

add SwissDict to docs

954b55d

eulerkochy changed the title ~~[WIP] SwissDict~~ SwissDict Aug 29, 2020

eulerkochy requested a review from oxinabox August 29, 2020 17:21

oxinabox reviewed Aug 30, 2020

View reviewed changes

oxinabox approved these changes Aug 30, 2020

View reviewed changes

eulerkochy added 2 commits August 30, 2020 21:33

changes because of using instead of import

ccf9ab5

fix tests, import length and isempty explicitly

12582d0

eulerkochy merged commit 9658145 into JuliaCollections:master Aug 30, 2020

PallHaraldsson mentioned this pull request Oct 25, 2020

Docs for SwissDict in wrong order, and questions about it #693

Open

tpapp mentioned this pull request Oct 26, 2020

Ordered dict based on SwissDict JuliaLang/julia#38167

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SwissDict #634

SwissDict #634

eulerkochy commented Jun 14, 2020 •

edited

oxinabox Jun 16, 2020

eulerkochy Jun 16, 2020

oxinabox Jun 16, 2020

oxinabox Jun 16, 2020

oxinabox Jun 16, 2020

oxinabox Jun 16, 2020

oxinabox Jun 16, 2020

oxinabox Jun 16, 2020

oxinabox Jun 16, 2020

rfourquet commented Jun 16, 2020

eulerkochy commented Jun 16, 2020

rfourquet commented Jun 16, 2020

vtjnash Jul 1, 2020

eulerkochy commented Aug 29, 2020 •

edited

oxinabox Aug 30, 2020

eulerkochy Aug 30, 2020


		#hashindex(key, sz) = (((hash(key)%Int) & (sz-1)) + 1)::Int

		##Simd utilities

SwissDict #634

SwissDict #634

Conversation

eulerkochy commented Jun 14, 2020 • edited

SwissDict

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfourquet commented Jun 16, 2020

eulerkochy commented Jun 16, 2020

rfourquet commented Jun 16, 2020

Choose a reason for hiding this comment

eulerkochy commented Aug 29, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eulerkochy commented Jun 14, 2020 •

edited

eulerkochy commented Aug 29, 2020 •

edited