error loading just-precompiled module if an imported module was loaded before precompiling

[dbrowne]

Getting the following error messages:

WARNING: could not import Base.help into PyCall
INFO: Precompiling module LaTeXStrings...
INFO: Recompiling stale cache file xxxxx/.julia/lib/v0.4/LaTeXStrings.ji for module LaTeXStrings.
WARNING: Module Compat uuid did not match cache file

LoadError: LoadError: __precompile__(true) but require failed to create a precompiled cache file
while loading xxxxxx/.julia/v0.4/PyPlot/src/PyPlot.jl, in expression starting on line 555
while loading In[2], in expression starting on line 1

 in require at ./loading.jl:211
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:263
 in require at ./loading.jl:202

---

Julia Version 0.4.0-dev+6526
Commit dcc448b* (2015-08-07 14:14 UTC)
Platform Info:
System: Linux (x86_64-redhat-linux)
CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
LAPACK: libopenblas
LIBM: libopenlibm
LLVM: libLLVM-3.3
Package Directory: xxxxxxxxxxxxxx/.julia/v0.4

WARNING: unknown Lazy commit 8709dc31, metadata may be ahead of package cache

15 required packages:

• ArgParse 0.2.11
• ArrayViews 0.6.2
• BayesNets 0.1.0
• CPUTime 0.0.4
• Cairo 0.2.28
• DataStructures 0.3.11
• Distributions 0.8.3
• Escher 0.1.0
• FactCheck 0.3.1
• GZip 0.2.17
• Gadfly 0.3.13
• IJulia 0.2.5
• Jewel 1.0.5
• PyPlot 1.5.3
• QuantEcon 0.2.0
  71 additional packages:
• Benchmark 0.1.0
• BinDeps 0.3.14
• Blosc 0.1.3
• Calculus 0.1.10
• Codecs 0.1.4
• Color 0.4.7
• Compat 0.4.12
• Compose 0.3.13
• Contour 0.0.7
• DSP 0.0.9
• DataArrays 0.2.17
• DataFrames 0.6.8
• Dates 0.4.4
• Distances 0.2.0
• Docile 0.5.14
• DualNumbers 0.1.3
• FixedPointNumbers 0.0.9
• Formatting 0.1.3
• FunctionalCollections 0.1.2
• GnuTLS 0.0.4
• Graphics 0.1.1
• Graphs 0.5.5
• Grid 0.3.10
• HDF5 0.5.2
• Hexagons 0.0.3
• Hiccup 0.0.0
• HttpCommon 0.0.12
• HttpParser 0.0.11
• HttpServer 0.0.12
• ImmutableArrays 0.0.9
• IterationManagers 0.0.1
• Iterators 0.1.8
• JLD 0.5.1
• JSON 0.4.4
• JuliaParser 0.6.2
• KernelDensity 0.1.1
• LNR 0.0.1
• LaTeXStrings 0.1.4
• Lazy 0.9.0+ master
• LightGraphs 0.2.2
• LightXML 0.1.13
• Loess 0.0.3
• MAT 0.2.12
• MacroTools 0.1.0
• Markdown 0.3.0
• Mux 0.0.0
• NaNMath 0.0.2
• Nettle 0.1.10
• Optim 0.4.2
• Options 0.2.4
• PDMats 0.3.5
• Patchwork 0.1.6
• Polynomials 0.0.3
• PyCall 0.8.2
• REPLCompletions 0.0.3
• Reactive 0.2.2
• Reexport 0.0.2
• Requires 0.1.3
• SHA 0.0.4
• Showoff 0.0.4
• SortingAlgorithms 0.0.5
• StatsBase 0.7.0
• StatsFuns 0.1.1
• TextWrap 0.1.3
• TikzGraphs 0.0.1
• TikzPictures 0.2.2
• URIParser 0.0.5
• WebSockets 0.0.6
• WoodburyMatrices 0.1.1
• ZMQ 0.1.20
• Zlib 0.1.8

[pao]

cc @stevengj (move to PyPlot/PyCall if this is package-specific and not a general problem)

[stevengj]

@dbrowne, had you just done Pkg.update()? What happens if you quit Julia and try again? Or try using LaTeXStrings?

[stevengj]

I think this might happen if Compat was already loaded (from using PyPlot, which calls using Compat), but not yet precompiled, before you first tried to precompile LaTeXStrings (which happens from the using LaTeXStrings in PyPlot).

When you compile LaTeXStrings, it also precompiles Compat.ji (using a separate Julia process that doesn't know that Compat was already loaded). Then it tries to load the LaTeXStrings module from the .ji file, but fails because the UUID of the Compat that was already loaded in memory doesn't match the UUID it expects (the one from Compat.ji on disk, which was not loaded because Compat was already imported).

[stevengj]

As a workaround, for now, using PyPlot should work if you restart your Julia session (since the second time around it will load the Compat.ji file from disk.

In the short term, we should probably patch Compat to include __precompile__(true) so that it is compiled the first time it is loaded, since it is safe to precompile anyway.

In the long term, two options are:

• Disallow auto-precompilation of a module unless all of its imported modules also have __precompile__(true).
• Precompiling a module Foo that imports Bar does not automatically precompile Bar unless Bar has __precompile__(true) also.

The second one seems more elegant to me, but is also more technically challenging since the precompile system now seems to assume that once you precompile you precompile all the way down. @vtjnash?

[vtjnash]

It's mostly a question of when you want your segfaults and incompatibilities. If you don't have tight control all of the way down, then you don't really know if some file was slightly different when it was originally loaded and might now be missing a definition which will cause the deserializer to segfault (you can try this at home by eval'ing a type or function into another module and then using that value locally -- see #12352

the first option is slightly annoying right now, but with precompile now being enforced by __precompile__, it's perhaps actually providing pretty good behavior.

[dbrowne]

If I quit Julia and try again I get the following:
WARNING: could not import Base.help into PyCall
WARNING: error initializing module PyPlot:
PyCall.PyError(msg=":PyImport_ImportModule", T=PyCall.PyObject(o=0x00007f8aa1ee4e60), val=PyCall.PyObject(o=0x0000000004213ea8), traceback=PyCall.PyObject(o=0x0000000000000000))
Attempting to plot returns this:
ERROR: UndefVarError: pltm not defined
in plot at /apps/infrafs1/dbrowne/.julia/v0.4/PyPlot/src/PyPlot.jl:368

[stevengj]

@dbrowne, the "error initializing module PyPlot" sounds like an unrelated problem—the PyImport_ImportModule exception means that libpython is failing to import matplotlib or some other necessary module. Please file a followup issue at PyPlot.

[dbrowne]

I raise the issue here because I'm building a new version of Julia and not PyPlot. If PyPlot were the moving target I would raise the issue there. Julia builds for the last few weeks worked without any issues.

[stevengj]

@dbrowne, the error message is different, you are seeing a different problem. Please use separate issues to discuss distinct problems.

[simonbyrne]

So what's the current recommended course of action? I'm trying to get tests working for a package that depends on DataFrames (which has a bunch of dependencies, not all of which have precompilation enabled).

[stevengj]

@simonbyrne, just import DataFrames first, and it will precompile all its dependencies.

The only problem arises if you manually import a dependency Foo that doesn't use __precompile__ first, and then import a package Bar that does use __precompile__(true) and which imports Foo. Then it runs into a problem because it wants the precompiled Foo but the non-precompiled Foo is already imported.

In the longer run, precompilable packages should push __precompile__() PRs to their dependencies. (Arguably, the best practice will be to push to your dependencies first, before adding __precompile__() to your own module.)

[simonbyrne]

Great, that fixed it, thanks!

[c42f]

A related problem appears when starting several separate julia processes in parallel. When they all hit the cache at once I get similar errors to the above.

(Background: the use case is an AWS cluster where instances are commonly started from scratch. When a node is first started the node downloads and installs a zip file containing a module with all the required julia dependencies. As soon as that's complete, many jobs jump onto it all at once and start hitting the compile cache, which basically guarantees a few failures as reproduced below.)

Simple test case involving DataFrames (run from a temporary directory):

$ cat init.sh

#!/bin/bash

pkgdir=$PWD/pkg
mkdir -p $pkgdir/v0.4

export JULIA_PKGDIR=$pkgdir
julia -e "Pkg.init()"
echo DataFrames > $pkgdir/v0.4/REQUIRE
julia -e "Pkg.resolve()"

$ cat test.sh

#!/bin/bash

pkgdir=$PWD/pkg
export JULIA_PKGDIR=$pkgdir
julia -e "using DataFrames"

Usage:

$ ./init.sh

# Now run two in parallel.
$ ./test.sh & ./test.sh 
[1] 13242
INFO: Recompiling stale cache file /home/chris/tmp/jlpkg/pkg/lib/v0.4/DataArrays.ji for module DataArrays.
WARNING: Module Reexport uuid did not match cache file
WARNING: require failed to create a precompiled cache file
WARNING: requiring "DataArrays" did not define a corresponding module.
ERROR: LoadError: UndefVarError: DataArrays not defined
 in include at ./boot.jl:260
 in include_from_node1 at ./loading.jl:271
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:284
 in _start at ./client.jl:411
while loading /home/chris/tmp/jlpkg/pkg/v0.4/DataFrames/src/DataFrames.jl, in expression starting on line 14
ERROR: Failed to precompile DataFrames to /home/chris/tmp/jlpkg/pkg/lib/v0.4/DataFrames.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:350
 in require at ./loading.jl:217
 in process_options at ./client.jl:284
 in _start at ./client.jl:411
INFO: Recompiling stale cache file /home/chris/tmp/jlpkg/pkg/lib/v0.4/DataFrames.ji for module DataFrames.

(julia version 0.4.0-pre+7400 commit bffe239)

[mlubin]

On a clean JuliaBox 0.4 notebook (no packages installed), I just ran Pkg.add("Cbc") and got the following:

INFO: Updating cache of BinDeps...
INFO: Cloning cache of Cbc from git://github.com/JuliaOpt/Cbc.jl.git
INFO: Updating cache of Compat...
INFO: Updating cache of MathProgBase...
INFO: Updating cache of SHA...
INFO: Updating cache of URIParser...
INFO: Installing BinDeps v0.3.18
INFO: Installing Cbc v0.1.8
INFO: Installing Compat v0.7.6
INFO: Installing MathProgBase v0.3.18
INFO: Installing SHA v0.1.2
INFO: Installing URIParser v0.1.1
INFO: Building Cbc
INFO: Precompiling module URIParser...
INFO: Precompiling module SHA...
INFO: Recompiling stale cache file /home/juser/.julia/lib/v0.4/Compat.ji for module Compat.
INFO: Recompiling stale cache file /home/juser/.julia/lib/v0.4/SHA.ji for module SHA.
WARNING: Module Compat uuid did not match cache file
=================================[ ERROR: Cbc ]=================================

LoadError: LoadError: LoadError: __precompile__(true) but require failed to create a precompiled cache file
while loading /home/juser/.julia/v0.4/BinDeps/src/dependencies.jl, in expression starting on line 969
while loading /home/juser/.julia/v0.4/BinDeps/src/BinDeps.jl, in expression starting on line 507
while loading /home/juser/.julia/v0.4/Cbc/deps/build.jl, in expression starting on line 1

================================================================================

================================[ BUILD ERRORS ]================================

Both Compat and SHA have precompile enabled. What's the issue?

[stevengj]

@mlubin, in a notebook, the IJulia package is loaded and hence Compat has already been loaded. Hence updating it runs into the same issue above.

[tkelman]

Note that #13506 cannot be backported until there is a followup that resolves the issues with Homebrew.jl and WinRPM.jl, both of which are is broken on 0.5. (winrpm may be working now - needs more investigation)

[Sisyphuss]

Just to report that I encountered recently similar problem with BinDeps, while building "IJulia" by Pkg.update(). I restarted Julia and then IJulia can be successfully built.

[stevengj]

@tkelman, did this ever get backported? I've been seeing reports of similar problems popping up recently, e.g. JuliaMath/Roots.jl#39

[tkelman]

was what backported? this issue was never closed.

[stevengj]

@tkelman, I was referring to #13458 #13506, which seems like it should have fixed most problems of this sort encountered in practice, e.g. the one in Roots.jl above.

[stevengj]

Oh, never mind, that was for Pkg.build. The precompilation is already done in separate processes. The original issue here still applies. Carry on then...

[vtjnash]

#18150 should make this work much better