Issue with parallel computation

[kyle11rd]

The following statement is posted under "High-Level Overview" section of the doc:

And I tried with the following funciton:

function test()
    addprocs(4)
    d = [1 2 1 4 5;3 2 1 6 5;3 1 1 7 4]
    som = initGigaSOM(d, 3, 3)
    som = trainGigaSOM(som, d)
    mapToGigaSOM(som, d)
    e = embedGigaSOM(som,d)
end

And lead to a problem with "worker 2":

ERROR: On worker 2:
KeyError: key GigaSOM [a03a9c34-069e-5582-a11c-5c984cab887c] not found
getindex at .\dict.jl:467 [inlined]
root_module at .\loading.jl:968 [inlined]
deserialize_module at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:953
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:855
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:773
deserialize_datatype at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:1251
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:826
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:773
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:833
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Serialization\src\Serialization.jl:773 [inlined]
deserialize_msg at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\messages.jl:99
#invokelatest#1 at .\essentials.jl:710 [inlined]
invokelatest at .\essentials.jl:709 [inlined]
message_handler_loop at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:185
process_tcp_streams at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:142
#99 at .\task.jl:356
Stacktrace:
 [1] #remotecall_fetch#143 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\remotecall.jl:394 [inlined]
 [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\remotecall.jl:386
 [3] #remotecall_fetch#146 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\remotecall.jl:421 [inlined]
 [4] remotecall_fetch at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\remotecall.jl:421 [inlined]
 [5] call_on_owner at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\remotecall.jl:494 [inlined]
 [6] fetch(::Future) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\remotecall.jl:533
 [7] distribute_array(::Symbol, ::Array{Float64,2}, ::Array{Int64,1}; dim::Int64) at C:\Users\zli3\.julia\packages\GigaSOM\QfzjJ\src\base\distributed.jl:88
 [8] distribute_array at C:\Users\zli3\.julia\packages\GigaSOM\QfzjJ\src\base\distributed.jl:78 [inlined]
 [9] trainGigaSOM(::Som, ::Array{Int64,2}; kernelFun::Function, metric::Distances.Euclidean, somDistFun::Function, knnTreeFun::Type{T} where T, rStart::Float64, rFinal::Float64, radiusFun::Function, epochs::Int64) at C:\Users\zli3\.julia\packages\GigaSOM\QfzjJ\src\analysis\core.jl:159
 [10] trainGigaSOM at C:\Users\zli3\.julia\packages\GigaSOM\QfzjJ\src\analysis\core.jl:156 [inlined]
 [11] test() at D:\...\runSOM.jl:27
 [12] top-level scope at none:1

Any thoughts?

[laurentheirendt]

The GigaSOM package must be loaded on every worker using the @everywhere macro:

using Distributed
addprocs(4)
@everywhere using GigaSOM

function test()
    # your test function content
end

test()

[kyle11rd]

Works! Thanks!!!

[laurentheirendt]

You are welcome 😄