Global and Distributed dictionaries for Julia.
GlobalDict and DistributedDict
This package provides two types of distributed dictionaries:
GlobalDictwhere the same key-value pair is stored on all participating workers.
DistributedDictwhere the key-value pairs are distributed over the participating workers based on the key's hash value.
GlobalDict(pids=PidsWorkers(); name::AbstractString=next_name(), ktype=Any, vtype=Any)where
pidsis one of
PidsAll()represents all processes in the cluster
PidsWorkers()represents all worker processes in the cluster
Pids([ids...])represents a specific subset of workers identified by their pids.
nameis a logical name for the dictionary. It should be unique across any such dictionary created across modules or different packages. Usually left unspecified in which case the system creates a unique name.
ktypeis the type of the keys. Defaults to
vtypeis the type of the values. Defaults to
DistributedDict(pids=PidsWorkers(); name::AbstractString=next_name(), ktype=Any, vtype=Any)similar to the
DistributedDict()only distribute over worker processes, i.e., excluding the master. Keyword args remain the same.
DistributedDict(pids::Array)distribute on the specified pids. Keyword args remain the same.
The following functions of
Associative are implemented:
Any closure wrapped in a ValueF will execute the closure on the node where the key is being assigned.
For example, for a
d[k] = ValueF(()->myid())
will set the value of d[k] to 2 on pid 2, 3 on pid 3 and so on.
And for a
DistributedDict, it will be set only on the node that the key hashes to.
The global and distributed dictionaries created by
DistributedDict are NOT collected when the objects go out of scope.
They have to be necessarily released by calling
Concurrent updates : This current version makes no attempt to handle race conditions when multiple tasks or processes update the same key. For example, if
d[k] = vsets
von all participating workers. If two tasks try to set
d[k] = v1and ``d[k] = v2
at the same time, there is no guarantee the value will be either allv1` or all `v2` on all workers.
Associativefunctions to be implemented.
Handle worker exits : Resiliency in the event of workers exiting has to be handled.