Large memory leak when using threads

[robsmith11]

With the following minimal example, I am seeing memory usage reach 10GB+ within seconds if I start Julia with JULIA_NUM_THREADS=4 (no other options).

If I only use single thread, memory usage remains around 200MB.

I've tested on both the 1.1 release and yesterday's nightly on a Linux Skylake Xeon.

Threads.@threads for i in 1:100000
    sum(collect(1:10^6))
end

[ChrisRackauckas]

Did you check if this is actually a leak by calling GC? A memory leak would mean that it doesn't get freed, not that the behavior of the GC might want to keep it around for a bit longer for performance reasons. Here, since the GC is not multithreaded, I would assume that the GC tries to not run for as long as possible when threaded, but that doesn't mean that your memory will fill up and Julia will crash. Instead it'll wait quite a bit until it either has to GC or it can GC after the multithreading, which is the behavior you would want for performance.

[chriselrod]

I tried:

julia> versioninfo()
Julia Version 1.3.0-DEV.163
Commit c40d9b099c (2019-05-04 03:30 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen Threadripper 1950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.0 (ORCJIT, znver1)
Environment:
  JULIA_NUM_THREADS = 16

julia> function foo(N)
           Threads.@threads for i in 1:N
               sum(collect(1:10^6))
           end
       end
foo (generic function with 1 method)

julia> @time foo(10^5)
 46.325609 seconds (337.58 k allocations: 736.214 GiB, 15.02% gc time)

julia> @time foo(10^5)
 42.238979 seconds (199.09 k allocations: 738.308 GiB, 9.36% gc time)

julia> GC.gc()

julia> @time foo(10^5)
 42.090349 seconds (199.11 k allocations: 738.472 GiB, 8.85% gc time)

So there seems to be a lot of GC activity. After the first run it used about 25G of RAM. After the last run, it was using over 32G.

[hycakir]

The issue is similar to the one in this SO question asked a while back. You do not need sum or anything else than allocations to create the leak.

function f(n) 
   Threads.@threads for i=1:n
        zeros(n)
   end
end

Try with large values like f(10^5) to see the issue. Forcing GC helps maybe a little.

[jpsamaroo]

For me, adding a call to GC.gc() within the @threads call at the end seems to keep memory usage in check, although things slow to a crawl. This is on commit 4dc15938bbb1b5f9fda9def3a85e80e3357a8193.

[jpsamaroo]

Maybe this is nothing, but when I run julia-debug under GDB, removing the call at the end to GC.gc() will cause sporadic segfaults on this line:

julia/base/locks-mt.jl

Line 44 in 9b5fea5

ccall(:jl_gc_safepoint, Cvoid, ())

I'm not on master, though, so I'll start a new build to see if this behaviour still occurs.

This is normal behavior (and also occurs on master).

[yuyichao]

The segfault is normal. You should ignore it.

[jpsamaroo]

I'm going to take a wild (uneducated) guess and say that somehow, the arrays getting allocated sometimes don't get rooted in the GC, and so effectively become lost (which is why an explicit GC.gc() after the @threads loop finished fails to collect the allocated arrays). To confirm this, I'm working on getting Valgrind to cooperate with me for this test case, but having some slight technical difficulties...

Assuming the above theory is plausible, is there any documentation on how I could go about recording GC statistics over time, e.g. to confirm that every allocation is later deallocated?

[yuyichao]

don't get rooted in the GC

No. Rooting does exactly the opposite.

[chethega]

Regarding whether the memory actually leaks: You need to test whether julia's memory use grows without bounds or reaches a steady state. With the default config, julia / linux / glibc is very bad at returning memory to the OS, probably due to heap fragmentation. However, the memory is still there and not leaked (glibc knows that the memory is free and will hand it out on malloc).

Has anyone tried reproducing on windows or osx?

[JeffBezanson]

Instead it'll wait quite a bit until it either has to GC or it can GC after the multithreading

This is incorrect; the GC does not wait for the end of @threads to run. It can run during the loop and is fully multi-threaded. The only capability we're missing is running julia code concurrently with GC, which is a very tall order. We can run non-julia (e.g. C) code concurrently with GC, and any julia code should at least handle threaded GC correctly.

[JeffBezanson]

I tried running #31923 (comment) on the 1.1 release binary and master and with 4 threads I see memory use holding steady at no more than about 3GB. On the second and third @time runs it goes up to 4-5GB but seems to eventually reach a steady state of about 4.3GB. Strange.

[dominikkiese]

Is there any progress on this subject? I'm running into similar issues with a simulation of mine. @code_warntype doesn't show any type instabilities but memory consumption fluctuates a lot after a few iterations even on current 1. 1 release.