NFFT in parallel

[DeVerMyst]

Hello, trying to update a software for Julia V.5 I got this problem:

nx = 64
nb = 195
nw = 20

serial

using NFFT

println(" serial ")
fr = rand(2,nb) - .5
plan = NFFTPlan( fr , (nx,nx))
x = rand(nx,nx) + 0.*im
F = nfft(plan,x)

parallel 1

println(" p1 ")
Fx = SharedArray( Complex{Float64}, (nb, nw))

@sync @parallel for n in 1:nw
println("worker")
plan = NFFTPlan( fr , (nx,nx))
x = rand(nx,nx) + 0.*im
Fx[:,n] = nfft(plan, x)
end

plan = NFFTPlan( fr , (nx,nx))
@sync @parallel for n in 1:nw
println("worker")
x = rand(nx,nx) + 0.*im
Fx[:,n] = nfft(plan, x)
end

julia> include("testnfft.jl")
serial
p1
From worker 2: worker

signal (11): Segmentation fault: 11
while loading no file, in expression starting on line 0
fftw_execute_dft at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia//libfftw3.dylib (unknown line)

• at ./fft/FFTW.jl:624

[robertdj]

On Julia v0.4.5 the code runs without errors. Let me find an v0.5 and test.

[robertdj]

I can run your code without errors in Julia v0.5.0 on Linux Mint. What does versioninfo() return on your computer?

[DeVerMyst]

Hello, I forgot to tell that I have problem with this code only when Julia is using several workers (julia -p nprocs). Similar code was working perfectly with the Version 4.5 of Julia.

julia> versioninfo()
Julia Version 0.5.0
Commit 3c9d753 (2016-09-19 18:14 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin13.4.0)
CPU: Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.7.1 (ORCJIT, sandybridge)

Regards

[robertdj]

I get seg faults on both Julia v0.4.5 and v0.5.0, but the fault is happening in fft (in Base Julia).

A recent change in NFFT is to use a precomputed plan to perform the ordinary FFT part: What used to be fft!(p.tmpVec) is now p.forwardFFT * p.tmpVec. Changing back to the old way makes your code run without errors.

I don't have experience with @sync and @parallel, but it seems weird to me that you can reuse the same NFFT plan in parallel loops: Each of them overwrites p.tmpVec for intermediate calculations. Have you verified that you get the correct results?
It seems to me that the only explanation is that multiple workers are not allowed to use the same ordinary FFT plan...

[robertdj]

Testing with the ordinary FFT confirms my suspicion: Precomputed plans for FFT do not work in parallel. Check e.g. the following:

A = rand(10, 4) + im
P = plan_fft( A[:,1] )
B = similar(A)

@sync @parallel for n in 1:4
    A_mul_B!( C[:,n], P, A[:,n] )
end

This seg faults my julia -p 2 in Julia v0.5.0.

It's unfortunate that the precomputed FFT plans in NFFT causes problems at this approach allows us to have only one nfft! method (and use less time/memory).

@tknopp : Any thoughts on this? Maybe it's better to ask wiser people for help with the FFT.

[tknopp]

If I remember correctly there is an option in FFTW to make a plan thread safe. This is a little different since it is about multi-process parallelism but it may work.

But our plan is certainly not safe to be used in different threads, because of the tmp vector.

[robertdj]

@DeVerMyst : Have you looked into the correctness of your results when using NFFT in parallel?

[DeVerMyst]

@robertdj : I use the NFFT in parallel in PAINTER.jl
Before making the choice of (or not) parallelizing some part of the algorithm I checked if results were equal (up to a small approximation) but in the case I am studying. So I can confirm that in the set of data I used to make the test, the results of NFFT in parallel were correct, and nicely fast.

[robertdj]

I have next to no experience with parallel computing in Julia, but I don't understand why the different threads are not overwriting the same temporary array...
We could revert to not using precomputed FFT plans, but I would like to figure out why this parallelizing works.

[tknopp]

@DeVerMyst: Its not surprising that your code works because in

https://github.com/andferrari/PAINTER.jl/blob/3a4dc61699ed23060422f6851085378a708b6f81/src/paintertools.jl#L62

you have n plans and use a dedicates plan for each process. So you have simply avoided that our API is not threadsafe.

[robertdj]

That explains it!

[DeVerMyst]

Hello,

Do you have any idea on how to make it work on julia v.5 ?

Regards

[tknopp]

@DeVerMyst your code should work on Julia 0.5. The example that you posted on top of this issue is not what you have done here:

https://github.com/andferrari/PAINTER.jl/blob/3a4dc61699ed23060422f6851085378a708b6f81/src/paintertools.jl#L62

As long as you have a dedicated plan for each process, it should work.

[DeVerMyst]

In order to present to You my problem I gave a lighter version of the code I use without thinking it can have a problem of temporary variable. I wrote to You because it was not working with Julia 0.5. Other libraries also, but I tried them one by one and I got this pb with NFFT and or FFTW.
For example this code which is totally equivalent to what is done in PAINTER.jl works with Julia 0.4.5 and not with Julia 0.5 with additional workers.

https://github.com/DeVerMyst/ForJulia5

Best regards

[tknopp]

That code works for me and does not segfault