Changes in memory allocation as a consequence of package precompilation

[timholy]

In JuliaGraphics/Colors.jl#370 it was noticed that adding precompile statements seems to have performance implications, and not always in a good way.

Running --track-allocation=all with the statements

julia> using Colors, FixedPointNumbers

julia> cols = rand(RGB{N0f8}, 10^5);

julia> cstrs = ['#'*hex(c) for c in cols];

julia> parsec(cstr) = parse(Colorant, cstr)
parsec (generic function with 1 method)

julia> map(parsec, cstrs);

julia> using Profile

julia> Profile.clear_malloc_data()

julia> map(parsec, cstrs);

and then moving the *.mem files (in base/ and all relevant packages) to /tmp/withpc (with precompilation) and /tmp/nopc (without precompilation) and running the following analysis script:

using Glob

jlfiles = Set{String}()
for dir in ["withpc", "nopc"]
    cd(dir) do
        for (root, dirs, files) in walkdir(".")
            for file in files
                push!(jlfiles, joinpath(root, splitext(splitext(file)[1])[1]))
            end
        end
    end
end


for file in jlfiles
    flw = glob(file*"*", "withpc")
    fln = glob(file*"*", "nopc")
    if length(flw) == length(fln) == 1
        fw, fn = normpath(flw[1]), normpath(fln[1])
        if !success(`cmp $fw $fn`)
            println(file, ':')
            run(ignorestatus(`diff -u --color $fw $fn`))
        end
    else
        println("file ", file, " is present only in ", isempty(flw) ? "nopc" : "withpc")
    end
end

yields a single relevant diff in Colors/parse.jl:

--- withpc/parse.jl.18420.mem	2019-12-09 02:37:18.800469670 -0600
+++ nopc/parse.jl.18505.mem	2019-12-09 02:39:45.171011466 -0600
@@ -59,7 +59,7 @@
         0     if mat != nothing
         0         prefix = mat.captures[1]
         0         len = length(mat.captures[2])
-  1599856         digits = parse(UInt32, mat.captures[2], base=16)
+        0         digits = parse(UInt32, mat.captures[2], base=16)
         0         if len == 6
   1600000             return convert(RGB{N0f8}, reinterpret(RGB24, digits))
         0         elseif len == 3

It is not clear to me why this line should allocate memory in one case but not the other. One can also verify that commenting out the 3 precompile statements for parse from the teh/precompile branch eliminates the extra allocation.

julia> versioninfo()
Julia Version 1.3.0
Commit 46ce4d7933* (2019-11-26 06:09 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIAFUNCDIR = /home/tim/juliafunc
  JULIA_CPU_THREADS = 4

[kimikage]

FYI, this may be related to the inlining. However, it is not clear which is the cause and which is the result.

Also, it seems that the behavior differs between the precompilation triggered by using and the manual Base.precompile calling. I think the difference is where they are executed, i.e. the "module".

[timholy]

As observed in #35972, this was fixed by #43990