Skip to content

Distributed table structures and data manipulation operations built on top of Dagger.jl

License

Notifications You must be signed in to change notification settings

JuliaParallel/DTables.jl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

69 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DTables

Distributed table structures and data manipulation operations built on top of Dagger.jl

Dev Build Status Coverage Code Style: Blue

Installation

The package registered in the general repository, so you can add it by typing:

julia> ]add DTables

Usage

Below you can find a quick example on how to get started with DTables.

There's a lot more you can do though, so please refer to the documentation!

# launch a Julia session with threads/workers

julia> using DTables

julia> dt = DTable((a=rand(100), b=rand(100)), 10)
DTable with 10 partitions
Tabletype: NamedTuple

julia> m = map(r -> (x=sum(r), id=Threads.threadid(),), dt)
DTable with 10 partitions
Tabletype: NamedTuple

julia> xsum = reduce((x, y) -> x + y, m, init=0, cols=[:x])
EagerThunk (running)

julia> threads_used = reduce((acc, el) -> union(acc, el), m, init=Set(), cols=[:id])
EagerThunk (running)

julia> fetch(xsum)
(x = 95.71209812014976,)

julia> fetch(threads_used)
(id = Set(Any[5, 4, 6, 13, 2, 10, 9, 12, 8, 3]),)

About

Distributed table structures and data manipulation operations built on top of Dagger.jl

Resources

License

Stars

Watchers

Forks

Languages