Memory used by dictionary never freed on Linux

[jekyllstein]

This issue was experienced on Linux with the following version info and is related to issue #25884 filed by @Mistguy :

Julia Version 1.0.3
Commit 1b59908 (2019-01-15 20:28 UTC)
Platform Info:
OS: Linux (x86_64-unknown-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

When a dictionary is created in a function or global space that has a matrix as its value, its memory never gets freed for the Linux OS kernel to use. I tested this in and out of interactive mode with the same effect.
I created the following script to test on different systems and saved it as dictAllocationTest.jl to run it as shown below:

using InteractiveUtils
versioninfo()

println()
if isinteractive()
	println("Julia running in interactive mode")
else
	println("Julia not running in interactive mode")
end
println()
totMem = Int64(Sys.total_memory())

println()
if isempty(ARGS)
	println("Testing in global space")
else
	println("Testing inside a function")
end
println()

function createTestDict(totMem)
	testDict = Dict{Int64, Matrix{Float64}}()
	for i in 1:100000
		push!(testDict, i => ones(Float64, 100, 100))
	end
	mem2 = totMem - Int64(Sys.free_memory())
	testDict = ()
	return mem2
end

mem1 = totMem - Int64(Sys.free_memory())

if isempty(ARGS)
	testDict = Dict{Int64, Matrix{Float64}}()
	for i in 1:100000
		push!(testDict, i => ones(Float64, 100, 100))
	end
	mem2 = totMem - Int64(Sys.free_memory())
	testDict = ()
else
	mem2 = createTestDict(totMem)
end

GC.gc()

mem3 = totMem - Int64(Sys.free_memory())

println("Memory added from dictionary is $((mem2 - mem1)/(10^9)) gigabytes")
println("Memory freed from clearing dictionary is $((mem2 - mem3)/(10^9)) gigabytes")
println("Final memory usage is $(100*mem3/mem1) % of the original")

Here are the results of testing it on linux in the global space:

> $julia1 dictAllocationTest.jl
Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode

Testing in global space
Memory added from dictionary is 8.016703488 gigabytes
Memory freed from clearing dictionary is 0.01390592 gigabytes
Final memory usage is 161.5230175061836 % of the original

and in the function space:

> $julia1 dictAllocationTest.jl jkl
Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode

Testing inside a function
Memory added from dictionary is 8.023105536 gigabytes
Memory freed from clearing dictionary is 0.0 gigabytes
Final memory usage is 161.65566657769227 % of the original

In both cases memory is not freed after calling GC.gc() even after the function call is completed. In contrast when I test this on macOS the results are very different:

$ $julia1 dictAllocationTest.jl
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode


Testing in global space

Memory added from dictionary is 8.097398784 gigabytes
Memory freed from clearing dictionary is 7.736045568 gigabytes
Final memory usage is 100.62443512513228 % of the original

$ $julia1 dictAllocationTest.jl jkgf
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia not running in interactive mode


Testing inside a function

Memory added from dictionary is 8.09453568 gigabytes
Memory freed from clearing dictionary is 7.783051264 gigabytes
Final memory usage is 100.53822059089042 % of the original

On macOS and Windows the memory is freed in both cases. It does seem however that if GC.gc() is called inside the function then memory is not freed until it is called after the function completes as I do in the script. At present I don't know how to free the memory without exiting the Julia session all together.

[nalimilan]

#25884 reports a similar issue involving Dict that happens on Linux but not on Windows.

[chethega]

Are you really sure that this is about freeing/leaking the memory, and not again about the glibc/malloc policy of when to remap / return memory to the kernel?

julia> @noinline function createTestDict(N)
               testDict = Dict{Int64, Matrix{Float64}}()
               for i in 1:N
                       push!(testDict, i => ones(Float64, 100, 100))
               end
               mem = Int64(Sys.free_memory())
               return mem
       end
julia> begin
       mem_init = Int64(Sys.free_memory())
       N=10_000
       for i=1:10
       if i <= 5 
          mem_ = createTestDict(N)
       else
          GC.gc()
          mem_ = Int64(Sys.free_memory())
       end
       println("run $i, use: $((mem_init - mem_) /1e9) GB")
       end
       end
run 1, use: 0.791433216 GB
run 2, use: 1.350914048 GB
run 3, use: 1.652056064 GB
run 4, use: 2.061836288 GB
run 5, use: 2.302337024 GB
run 6, use: 0.87791616 GB
run 7, use: 0.87791616 GB
run 8, use: 0.878174208 GB
run 9, use: 0.878174208 GB
run 10, use: 0.878174208 GB

with

$ uname -rvm
4.20.0-arch1-1-ARCH #1 SMP PREEMPT Mon Dec 24 03:00:40 UTC 2018 x86_64
$ ldd --version
ldd (GNU libc) 2.28

[jekyllstein]

Matching your test above if I keep defining a new Dict object the memory keeps accumulating until GC is called. After that it will clear all the memory allocated to the Dicts except the original one which remains in memory when seems to match your test. Also in tests where I've deleted entries from an existing dictionary the memory use doesn't go down at all on linux.

[jekyllstein]

Test Script

using InteractiveUtils
versioninfo()

println()
if isinteractive()
	println("Julia running in interactive mode")
else
	println("Julia not running in interactive mode")
end
println()
totMem = Int64(Sys.total_memory())

@noinline function createTestDict(N)
	testDict = Dict{Int64, Matrix{Float64}}()
	for i in 1:N
		push!(testDict, i => ones(Float64, 100, 100))
	end
	(testDict, Int64(Sys.free_memory()))
end

@noinline function growTestDict!(testDict, N1, N2)
	for i in N1+1:N2
		push!(testDict, i => ones(Float64, 100, 100))
	end
	Int64(Sys.free_memory())
end

@noinline function shrinkTestDict!(testDict, N)
	ks = collect(keys(testDict))[1:N]
	for k in ks
		delete!(testDict, k)
	end
	Int64(Sys.free_memory())
end

function modifyDict!(testDict, N)
	for i in 1:10
		mem = growTestDict!(testDict, i*N, (i+1)*N)
		println("Memory use after growing dictionary is $((mem1 - mem)/(10^9)) gigabytes")
		mem = shrinkTestDict!(testDict, N)
		GC.gc()
		println("Memory use after shrinking dictionary is $((mem1 - mem)/(10^9)) gigabytes")
	end
end

function runTest(mem1)

	N = 10000
	(testDict, mem2) = createTestDict(N)

	println("Memory added from dictionary is $((mem1 - mem2)/(10^9)) gigabytes")

	modifyDict!(testDict, N)
end

mem1 = Int64(Sys.free_memory())
runTest(mem1)
GC.gc()
println("Final memory use is $((mem1 - Int64(Sys.free_memory()))/(10^9)) gigabytes")

Results

Save the above script as dictGrowShrinkTest.jl and running twice in interactive mode produced the following results for me on different systems:

Windows 10

julia> include("dictGrowShrinkTest.jl")
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD FX(tm)-6300 Six-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, bdver1)

Julia running in interactive mode

Memory added from dictionary is 0.787034112 gigabytes
Memory use after growing dictionary is 1.584635904 gigabytes
Memory use after shrinking dictionary is 1.584635904 gigabytes
Memory use after growing dictionary is 1.582608384 gigabytes
Memory use after shrinking dictionary is 1.582620672 gigabytes
Memory use after growing dictionary is 1.580736512 gigabytes
Memory use after shrinking dictionary is 1.580867584 gigabytes
Memory use after growing dictionary is 1.456144384 gigabytes
Memory use after shrinking dictionary is 1.456144384 gigabytes
Memory use after growing dictionary is 1.454751744 gigabytes
Memory use after shrinking dictionary is 1.454993408 gigabytes
Memory use after growing dictionary is 1.462607872 gigabytes
Memory use after shrinking dictionary is 1.462603776 gigabytes
Memory use after growing dictionary is 1.46192384 gigabytes
Memory use after shrinking dictionary is 1.461919744 gigabytes
Memory use after growing dictionary is 1.458970624 gigabytes
Memory use after shrinking dictionary is 1.459032064 gigabytes
Memory use after growing dictionary is 1.459163136 gigabytes
Memory use after shrinking dictionary is 1.459204096 gigabytes
Memory use after growing dictionary is 1.459593216 gigabytes
Memory use after shrinking dictionary is 1.459654656 gigabytes
Final memory use is -0.122564608 gigabytes

julia> include("dictGrowShrinkTest.jl")
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD FX(tm)-6300 Six-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, bdver1)

Julia running in interactive mode

Memory added from dictionary is 0.799145984 gigabytes
Memory use after growing dictionary is 1.586065408 gigabytes
Memory use after shrinking dictionary is 1.58806016 gigabytes
Memory use after growing dictionary is 1.589088256 gigabytes
Memory use after shrinking dictionary is 1.590575104 gigabytes
Memory use after growing dictionary is 1.588686848 gigabytes
Memory use after shrinking dictionary is 1.588670464 gigabytes
Memory use after growing dictionary is 1.5325184 gigabytes
Memory use after shrinking dictionary is 1.532588032 gigabytes
Memory use after growing dictionary is 1.53051136 gigabytes
Memory use after shrinking dictionary is 1.530494976 gigabytes
Memory use after growing dictionary is 1.529745408 gigabytes
Memory use after shrinking dictionary is 1.529729024 gigabytes
Memory use after growing dictionary is 1.524752384 gigabytes
Memory use after shrinking dictionary is 1.524752384 gigabytes
Memory use after growing dictionary is 1.52492032 gigabytes
Memory use after shrinking dictionary is 1.52492032 gigabytes
Memory use after growing dictionary is 1.52498176 gigabytes
Memory use after shrinking dictionary is 1.525112832 gigabytes
Memory use after growing dictionary is 1.526677504 gigabytes
Memory use after shrinking dictionary is 1.5266816 gigabytes
Final memory use is -0.065601536 gigabytes

MacOSX

julia> include("dictGrowShrinkTest.jl")
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, ivybridge)

Julia running in interactive mode

Memory added from dictionary is 0.805318656 gigabytes
Memory use after growing dictionary is 1.613824 gigabytes
Memory use after shrinking dictionary is 1.613828096 gigabytes
Memory use after growing dictionary is 1.628459008 gigabytes
Memory use after shrinking dictionary is 1.628459008 gigabytes
Memory use after growing dictionary is 1.64102144 gigabytes
Memory use after shrinking dictionary is 1.64102144 gigabytes
Memory use after growing dictionary is 1.64102144 gigabytes
Memory use after shrinking dictionary is 1.64102144 gigabytes
Memory use after growing dictionary is 1.647296512 gigabytes
Memory use after shrinking dictionary is 1.647296512 gigabytes
Memory use after growing dictionary is 1.653551104 gigabytes
Memory use after shrinking dictionary is 1.653551104 gigabytes
Memory use after growing dictionary is 1.653530624 gigabytes
Memory use after shrinking dictionary is 1.653530624 gigabytes
Memory use after growing dictionary is 1.653530624 gigabytes
Memory use after shrinking dictionary is 1.653530624 gigabytes
Memory use after growing dictionary is 1.6507904 gigabytes
Memory use after shrinking dictionary is 1.6507904 gigabytes
Memory use after growing dictionary is 1.6507904 gigabytes
Memory use after shrinking dictionary is 1.6507904 gigabytes
Final memory use is 0.219844608 gigabytes

julia> include("dictGrowShrinkTest.jl")
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, ivybridge)

Julia running in interactive mode

Memory added from dictionary is 0.620756992 gigabytes
Memory use after growing dictionary is 1.428160512 gigabytes
Memory use after shrinking dictionary is 1.428160512 gigabytes
Memory use after growing dictionary is 1.432354816 gigabytes
Memory use after shrinking dictionary is 1.432354816 gigabytes
Memory use after growing dictionary is 1.439719424 gigabytes
Memory use after shrinking dictionary is 1.439719424 gigabytes
Memory use after growing dictionary is 1.439711232 gigabytes
Memory use after shrinking dictionary is 1.439711232 gigabytes
Memory use after growing dictionary is 1.443872768 gigabytes
Memory use after shrinking dictionary is 1.443872768 gigabytes
Memory use after growing dictionary is 1.448067072 gigabytes
Memory use after shrinking dictionary is 1.448067072 gigabytes
Memory use after growing dictionary is 1.448067072 gigabytes
Memory use after shrinking dictionary is 1.448067072 gigabytes
Memory use after growing dictionary is 1.452244992 gigabytes
Memory use after shrinking dictionary is 1.452244992 gigabytes
Memory use after growing dictionary is 1.452244992 gigabytes
Memory use after shrinking dictionary is 1.452244992 gigabytes
Memory use after growing dictionary is 1.452244992 gigabytes
Memory use after shrinking dictionary is 1.452244992 gigabytes
Final memory use is 0.013062144 gigabytes

SUSE Linux

julia> include("dictGrowShrinkTest.jl")
Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia running in interactive mode

Memory added from dictionary is 0.794963968 gigabytes
Memory use after growing dictionary is 1.598976 gigabytes
Memory use after shrinking dictionary is 1.598976 gigabytes
Memory use after growing dictionary is 1.96847616 gigabytes
Memory use after shrinking dictionary is 1.96847616 gigabytes
Memory use after growing dictionary is 2.09291264 gigabytes
Memory use after shrinking dictionary is 2.093420544 gigabytes
Memory use after growing dictionary is 2.15386112 gigabytes
Memory use after shrinking dictionary is 2.15386112 gigabytes
Memory use after growing dictionary is 2.189930496 gigabytes
Memory use after shrinking dictionary is 2.189930496 gigabytes
Memory use after growing dictionary is 2.21837312 gigabytes
Memory use after shrinking dictionary is 2.218881024 gigabytes
Memory use after growing dictionary is 2.278375424 gigabytes
Memory use after shrinking dictionary is 2.278375424 gigabytes
Memory use after growing dictionary is 2.288922624 gigabytes
Memory use after shrinking dictionary is 2.288922624 gigabytes
Memory use after growing dictionary is 2.31000064 gigabytes
Memory use after shrinking dictionary is 2.310508544 gigabytes
Memory use after growing dictionary is 2.310508544 gigabytes
Memory use after shrinking dictionary is 2.310508544 gigabytes
Final memory use is 2.0509696 gigabytes

julia> include("dictGrowShrinkTest.jl")
Julia Version 1.0.3
Commit 1b5990863d (2019-01-15 20:28 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

Julia running in interactive mode

Memory added from dictionary is -0.121888768 gigabytes
Memory use after growing dictionary is 0.283291648 gigabytes
Memory use after shrinking dictionary is 0.283291648 gigabytes
Memory use after growing dictionary is 0.683012096 gigabytes
Memory use after shrinking dictionary is 0.68352 gigabytes
Memory use after growing dictionary is 0.794750976 gigabytes
Memory use after shrinking dictionary is 0.794750976 gigabytes
Memory use after growing dictionary is 0.837398528 gigabytes
Memory use after shrinking dictionary is 0.837906432 gigabytes
Memory use after growing dictionary is 0.85415936 gigabytes
Memory use after shrinking dictionary is 0.85415936 gigabytes
Memory use after growing dictionary is 0.872316928 gigabytes
Memory use after shrinking dictionary is 0.872316928 gigabytes
Memory use after growing dictionary is 0.872316928 gigabytes
Memory use after shrinking dictionary is 0.872316928 gigabytes
Memory use after growing dictionary is 0.87257088 gigabytes
Memory use after shrinking dictionary is 0.87257088 gigabytes
Memory use after growing dictionary is 0.87257088 gigabytes
Memory use after shrinking dictionary is 0.87257088 gigabytes
Memory use after growing dictionary is 0.878649344 gigabytes
Memory use after shrinking dictionary is 0.878649344 gigabytes
Final memory use is -0.121667584 gigabytes

Comments

On macOS and Windows the memory is mostly freed after the function call finishes and within each step memory use doesn't exceed the original "grow" allocation. In Linux on the first run the memory seems to grow every step even though the dictionary size stays constant and the memory is not freed when the function completes either. But on the second pass the system does seem to take advantage of memory that was already allocated. Idk why the behavior is like this and what accounts for the variation between systems but once you make those initial allocations on Linux I can't find a way to get rid of them. @chethega this Linux system uses an older version of libc

shell> ldd --version
ldd (GNU libc) 2.11.3

so I'm curious how this type of test works on your system considering its similarities with the other one you posted.

[JeffBezanson]

Maybe a duplicate of #30653?

[jekyllstein]

Maybe a duplicate of #30653?

They might be related. I don't know enough about GC to know. Do you know if that other issue has been fixed and if so which julia version I'd test to confirm that?

[JeffBezanson]

The fix for that is on master, if you're able to try it.

[KristofferC]

It is also in the 1.2-rc2 which can be downloaded at https://julialang.org/downloads/.