Memory leak when creating plots #726

Octogonapus · 2020-10-15T14:35:45Z

Describe the bug

When creating a plot (e.g., scatter(1:20000)), the memory used for that plot is retained indefinitely and never reclaimed by the GC.

Steps To Reproduce

These two examples both retain around 15GB of memory forever:

Example 1

using AbstractPlotting, GLMakie
plot = scatter(1:2000000000);
GLMakie.destroy!.(plot.current_screens)
plot = nothing
AbstractPlotting.current_global_scene[] = nothing
GC.gc()

Example 2

using AbstractPlotting, GLMakie

function do_leak()
    plot = scatter(1:200000000)
    GLMakie.destroy!.(plot.current_screens)
    plot = nothing
    AbstractPlotting.current_global_scene[] = nothing
    GC.gc()
end

for i in 1:10
    do_leak()
end

Expected behavior

After I create a plot and discard all references to it that I know of (i.e., the plot reference itself and AbstractPlotting.current_global_scene), I expect the memory used for that plot to be freed.

Additional Information

This looks similar to #382

This has been reproduced by @ianshmean

This is the output from varinfo: https://pastebin.com/qAy0Zvtj

Here is a more descriptive example and the outputs it produces:

using AbstractPlotting, GLMakie

function do_leak()
    print("Start of do_leak: ")
    @show Base.summarysize(AbstractPlotting)

    plot = scatter(1:200000000)
    print("Plot: ")
    @show Base.summarysize(plot)
    
    GLMakie.destroy!.(plot.current_screens)
    print("Plot after GLMakie.destroy!: ")
    @show Base.summarysize(plot)
    
    plot = nothing

    print("AbstractPlotting.current_global_scene after GLMakie.destroy!: ")
    @show Base.summarysize(AbstractPlotting.current_global_scene)
    AbstractPlotting.current_global_scene[] = nothing
    
    GC.gc()
    
    print("After GC: ")
    @show Base.summarysize(AbstractPlotting)
end

for i in 1:10
    println("Iteration $i")
    do_leak()
    println()
end

Iteration 1
Start of do_leak: Base.summarysize(AbstractPlotting) = 3280498
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 1611982740

Iteration 2
Start of do_leak: Base.summarysize(AbstractPlotting) = 1611982740
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 3212083673

Iteration 3
Start of do_leak: Base.summarysize(AbstractPlotting) = 3212083673
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 4812184606

Iteration 4
Start of do_leak: Base.summarysize(AbstractPlotting) = 4812184606
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 6412285539

Iteration 5
Start of do_leak: Base.summarysize(AbstractPlotting) = 6412285539
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 8012386472

Iteration 6
Start of do_leak: Base.summarysize(AbstractPlotting) = 8012386472
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 9612487405

Iteration 7
Start of do_leak: Base.summarysize(AbstractPlotting) = 9612487405
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 11212588338

Iteration 8
Start of do_leak: Base.summarysize(AbstractPlotting) = 11212588338
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 12812689271

Iteration 9
Start of do_leak: Base.summarysize(AbstractPlotting) = 12812689271
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 14412790204

Iteration 10
Start of do_leak: Base.summarysize(AbstractPlotting) = 14412790204
Plot: Base.summarysize(plot) = 1600099693
Plot after GLMakie.destroy!: Base.summarysize(plot) = 1600099693
AbstractPlotting.current_global_scene after GLMakie.destroy!: Base.summarysize(AbstractPlotting.current_global_scene) = 1600099701
After GC: Base.summarysize(AbstractPlotting) = 16012891137

The text was updated successfully, but these errors were encountered:

Octogonapus · 2020-10-15T15:17:18Z

With a patched varinfo to show everything (including non-exported names and imported names), I see these relevant lines:

| AbstractPlotting              |  14.913 GiB | Module    
| _current_default_theme        |  14.902 GiB | Attributes     
| minimal_default               |  14.902 GiB | Attributes

The sizes of the members of that Dict:

Base.summarysize(AbstractPlotting._current_default_theme[colormap]) = 4376
Base.summarysize(AbstractPlotting._current_default_theme[visible]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[axis]) = 464
Base.summarysize(AbstractPlotting._current_default_theme[axis2d]) = 464
Base.summarysize(AbstractPlotting._current_default_theme[markersize]) = 4864
Base.summarysize(AbstractPlotting._current_default_theme[camera]) = 4376
Base.summarysize(AbstractPlotting._current_default_theme[linestyle]) = 4376
Base.summarysize(AbstractPlotting._current_default_theme[font]) = 16000983518
Base.summarysize(AbstractPlotting._current_default_theme[axis_type]) = 4376
Base.summarysize(AbstractPlotting._current_default_theme[raw]) = 3657
Base.summarysize(AbstractPlotting._current_default_theme[clear]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[legend]) = 464
Base.summarysize(AbstractPlotting._current_default_theme[axis3d]) = 464
Base.summarysize(AbstractPlotting._current_default_theme[resolution]) = 16000983694
Base.summarysize(AbstractPlotting._current_default_theme[update_limits]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[center]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[marker]) = 4416
Base.summarysize(AbstractPlotting._current_default_theme[color]) = 4376
Base.summarysize(AbstractPlotting._current_default_theme[show_legend]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[padding]) = 5108
Base.summarysize(AbstractPlotting._current_default_theme[SSAO]) = 14084
Base.summarysize(AbstractPlotting._current_default_theme[palette]) = 18416
Base.summarysize(AbstractPlotting._current_default_theme[backgroundcolor]) = 4376
Base.summarysize(AbstractPlotting._current_default_theme[show_axis]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[scale_plot]) = 4377
Base.summarysize(AbstractPlotting._current_default_theme[limits]) = 16000983518

julia> AbstractPlotting._current_default_theme[:font]
Observable{Any} with 20 listeners. Value:
"Dejavu Sans"

I suppose that the observable may, through the listeners, work its way back to all the plots.

jkrumbiegel · 2020-10-15T16:14:30Z

And this is exactly the reason why I was pushing for better options for disconnecting things in Makie / Observables. Because if everybody just hooks into these global observables without cleanup, references to almost everything stay in memory. I think a lot of this requires some sort of redesign to make it easier to disconnect.

This JuliaGizmos/Observables.jl#48 was part of what I tried to alleviate the issue, but there was only so much you could do without breaking Observables or Makie.

SimonDanisch · 2020-10-15T16:19:47Z

That's true, that it's part of the problem...
But something also must have change here, since this is clearly a regression, as far as I can tell...
It's also relatively weird, that resolution ends up having a connection to the scatter input^^

Octogonapus · 2020-10-15T17:54:23Z

In the meantime, is there a recommended workaround?

I will do this unless someone has a better suggestion:

for k in keys(AbstractPlotting._current_default_theme)
    try
        empty!(AbstractPlotting._current_default_theme[k].listeners)
    catch ex
        if !isa(ex, KeyError)
            throw(ex)
        end
    end
end

Octogonapus · 2020-12-01T15:25:02Z

@SimonDanisch Thanks for your work on this issue so far. It looks like this isn't fixed yet, though. Using AbstractPlotting v0.13.10 and GLMakie v0.1.16, this code:

using AbstractPlotting, GLMakie

function do_leak()
    plot = scatter(1:200000000)
    GLMakie.destroy!.(plot.current_screens)
    plot = nothing
    AbstractPlotting.current_global_scene[] = nothing
    GC.gc(true)
    @info "Size of AbstractPlotting: $(Base.format_bytes(Base.summarysize(AbstractPlotting)))"
end

for i in 1:10
    @info "Iteration $i"
    do_leak()
    println()
end

outputs this:

[ Info: Iteration 1
[ Info: Size of AbstractPlotting: 1.501 GiB

[ Info: Iteration 2
[ Info: Size of AbstractPlotting: 2.992 GiB

[ Info: Iteration 3
[ Info: Size of AbstractPlotting: 4.482 GiB

[ Info: Iteration 4
[ Info: Size of AbstractPlotting: 5.972 GiB

[ Info: Iteration 5
[ Info: Size of AbstractPlotting: 7.462 GiB

[ Info: Iteration 6
[ Info: Size of AbstractPlotting: 8.952 GiB

[ Info: Iteration 7
[ Info: Size of AbstractPlotting: 10.443 GiB

[ Info: Iteration 8
[ Info: Size of AbstractPlotting: 11.933 GiB

[ Info: Iteration 9
[ Info: Size of AbstractPlotting: 13.423 GiB

[ Info: Iteration 10
[ Info: Size of AbstractPlotting: 14.913 GiB

Let me know if you want me to test anything else.

fix MakieOrg/Makie.jl#726 for real

SimonDanisch added a commit to JuliaPlots/AbstractPlotting.jl that referenced this issue Nov 16, 2020

fix memory leak MakieOrg/Makie.jl#726

f1ce077

SimonDanisch closed this as completed in JuliaPlots/AbstractPlotting.jl@0b8067b Dec 11, 2020

SimonDanisch added a commit to JuliaPlots/AbstractPlotting.jl that referenced this issue Dec 11, 2020

Merge pull request #574 from JuliaPlots/sd/mem_leak_v2

8b81fe8

fix MakieOrg/Makie.jl#726 for real

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak when creating plots #726

Memory leak when creating plots #726

Octogonapus commented Oct 15, 2020 •

edited

Octogonapus commented Oct 15, 2020 •

edited

jkrumbiegel commented Oct 15, 2020

SimonDanisch commented Oct 15, 2020

Octogonapus commented Oct 15, 2020 •

edited

Octogonapus commented Dec 1, 2020

Memory leak when creating plots #726

Memory leak when creating plots #726

Comments

Octogonapus commented Oct 15, 2020 • edited

Describe the bug

Steps To Reproduce

Example 1

Example 2

Expected behavior

Additional Information

Octogonapus commented Oct 15, 2020 • edited

jkrumbiegel commented Oct 15, 2020

SimonDanisch commented Oct 15, 2020

Octogonapus commented Oct 15, 2020 • edited

Octogonapus commented Dec 1, 2020

Octogonapus commented Oct 15, 2020 •

edited

Octogonapus commented Oct 15, 2020 •

edited

Octogonapus commented Oct 15, 2020 •

edited