parallelization results in signal(11): Segmentation fault

[andrewkhardy]

I am suffering from a segmentation fault 11 error when multithreading using joblib.Parallel to run a small Julia function.

I import a function in Python, modeled after this attempted solution

from julia.api import Julia
jl = Julia(compiled_modules=False)
from julia import Main
Main.include("fastsum.jl")
from julia.Main import greenfunction

where the function itself is

using Tullio, LoopVectorization
function greenfunction(mu, wns, sigwns, energy, dosnorm)
    new_array = Array{ComplexF64}(undef, length(wns))
    @tullio threads=false new_array[i] = 1 / (mu + wns[i] * 1im - energy[j] - sigwns[i]) * dosnorm[j]
    dosnorm = Nothing
    energy  = Nothing
    sigwns  = Nothing
    wns     = Nothing
    
    return new_array
end

Clearing some of the variables seems to make the number of processes I can run increase from something around 10 to 80, but eventually it still segfaults, with error as

signal (15): Terminated
in expression starting at none:0
mul_fast at ./fastmath.jl:167 [inlined]
mul_fast at ./fastmath.jl:219 [inlined]
...
... ( a lot of PyCall directories)
...
unknown function (ip: (nil))
Allocations: 76861098 (Pool: 76845028; Big: 16070); GC: 116

signal (11): Segmentation fault
in expression starting at none:0
...
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The more detailed errors are here

Is there a way to clear the Julia cache as this seems to be the problem? Or import Julia less frequently somehow? I call that import function several hundred times.

I also attempted

from julia import Main
greenfunction = Main.eval("""
using Tullio, LoopVectorization
function greenfunction(mu, wns, sigwns, energy, dosnorm)
    new_array = Array{ComplexF64}(undef, length(wns))
    @tullio threads=false new_array[i] = 1 / (mu + wns[i] * 1im - energy[j] - sigwns[i]) * dosnorm[j]
    dosnorm = Nothing
    energy  = Nothing
    sigwns  = Nothing
    wns     = Nothing
    
    return new_array
end
""")

but that gives the same error.

[fchorney]

I am having similar issues. I am trying to use pyjulia in a flask web server but it just keeps segfaulting as I assume the webserver is multithreaded.

[00sapo]

Try using w+ mode for memmapping (see #472)

[sibyjackgrove]

I think this is related to #477 and SciML/diffeqpy#98

@andrewkhardy Did you find any solution?

[tkf]

IIUC, the default backend of joblib is the loky package that uses fork on POSIX system. However, Julia is not fork-safe. So, it seems like you need to use the "multiprocessing" backend in joblib and then call multiprocessing.set_start_method("spawn") to use the spawn method in multiprocessing (Related discussion: https://discourse.julialang.org/t/calling-julia-from-python-on-thread/43187/3).

[sibyjackgrove]

@tkf I am not using the multiprocessing module. I am using the subprocess module to launch my Python sub-processes. Will this fix work with that case?

[tkf]

What subprocess.Popen uses is not very clear from the documentation. Somebody has to dig into the internal of subprocess.