Intermittently broken precompile #13200

yuyichao · 2015-09-18T04:11:26Z

Update: Just to be clear, this issue doesn't seem to be OSX specific althought it does happen more often on OSX travis.

Original post:

This started to happen on the OSX Travis CI since a few days ago for both 0.4 and master. (Also possible that the OSX travis was just not enabled before that). This is a typical failed build (output attached in case someone restarted that build).

The corresponding Linux build looks fine.

INFO: Package database updated
INFO: Recompiling stale cache file /Users/travis/.julia/lib/v0.4/Requests.ji for module Requests.
WARNING: Module JSON uuid did not match cache file
ERROR: LoadError: LoadError: __precompile__(true) but require failed to create a precompiled cache file
 in require at /Users/travis/julia/lib/julia/sys.dylib
 in include at /Users/travis/julia/lib/julia/sys.dylib
 in include_from_node1 at /Users/travis/julia/lib/julia/sys.dylib
 in include at /Users/travis/julia/lib/julia/sys.dylib
 in include_from_node1 at /Users/travis/julia/lib/julia/sys.dylib
 in require at /Users/travis/julia/lib/julia/sys.dylib
 in process_options at /Users/travis/julia/lib/julia/sys.dylib
 in _start at /Users/travis/julia/lib/julia/sys.dylib
while loading /Users/travis/.julia/v0.4/Coverage/src/coveralls.jl, in expression starting on line 5
while loading /Users/travis/.julia/v0.4/Coverage/src/Coverage.jl, in expression starting on line 185

Keno · 2015-09-18T04:50:32Z

I've seen this kind of thing several times (on OS X). Usually it's fine the next time I start julia though.

yuyichao · 2015-09-19T01:11:09Z

Ref #13182 (comment) This might not be OSX specific (but it only happens on Travis OSX and not Travis Linux AFAICT)

yuyichao · 2015-09-19T17:10:20Z

I've just seen this locally as well. It happend when I run Pkg.test("Cairo") after removing all the cache files.

jiahao · 2015-09-19T17:13:22Z

It can be reproduced ~50% of the time running the World of Julia script.

tkelman · 2015-09-24T04:09:54Z

This happens 100% of the time if you run vagrant up nightlyMZ in PackageEvaluator with 0.4.0-rc1 or newer. Badly needs fixing. Happens after running tests for MacroTools.jl. Unrelated to that package afaict:

==> nightlyMZ: ERROR: LoadError: ArgumentError: DataStructures not found in path
==> nightlyMZ:  in require at ./loading.jl:252
==> nightlyMZ:  in stale_cachefile at loading.jl:439
==> nightlyMZ:  in recompile_stale at loading.jl:457
==> nightlyMZ:  in _require_from_serialized at loading.jl:83
==> nightlyMZ:  in _require_from_serialized at ./loading.jl:109
==> nightlyMZ:  in require at ./loading.jl:219
==> nightlyMZ:  in include at ./boot.jl:261
==> nightlyMZ:  in include_from_node1 at ./loading.jl:304
==> nightlyMZ:  in process_options at ./client.jl:308
==> nightlyMZ:  in _start at ./client.jl:411
==> nightlyMZ: while loading /home/vagrant/pkgeval/src/prepjson.jl, in expression starting on line 11
==> nightlyMZ: INFO: Removing MacroTools v0.2.0
==> nightlyMZ: INFO: Package database updated

edit: this is actually probably a JSON.jl issue, not a base issue: JuliaIO/JSON.jl#123 (comment)

Keno · 2015-09-26T19:58:05Z

I just saw this, not sure if it's related, but seems bad as well:

julia> using Plots
[Plots.jl] Default backend: pyplot

julia> using GtkInteract
ERROR: LoadError: MethodError: `module_uuid` has no method matching module_uuid(::Type{ColorTypes.Color{T,N}})
 in stale_cachefile at ./loading.jl:441
 in recompile_stale at ./loading.jl:457
 in _require_from_serialized at ./loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:219
 in stale_cachefile at ./loading.jl:439
 in recompile_stale at ./loading.jl:457
 in _require_from_serialized at ./loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:219
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in require at ./loading.jl:243
while loading /Users/kfischer/.julia/v0.5/GtkInteract/src/GtkInteract.jl, in expression starting on line 12

julia>

timholy · 2015-09-26T22:56:57Z

That error means that you've got some conflict between Colors and Color somewhere on your system: something is expecting a module named Color, but in reality it's a type.

tkelman · 2015-10-03T16:52:01Z

I got this to happen fairly reliably when doing Pkg.add("GeoIP"); Pkg.test("GeoIP") starting from a clean no-packages state on linux64

mdcfrancis · 2016-01-13T16:51:11Z

My issue which was closed as a dupe includes a reproducible (simple) example which depends on no external packages.

#14663

The problem occurs when a dependent module 'uses' a package which does not have precompile set and then you use this module. This only occurs first time round with an empty lib directory.

vtjnash · 2016-01-13T17:10:34Z

that is simply a usage error. __precompile__(true) is a user-annotation stating that all of the preconditions for compilation are going to be met. one of those conditions is that all modules that it uses also has that flag set. the system doesn't have a way to check many of the preconditions, but they will cause various other parts of the system to fail. documentation for this should be added to http://docs.julialang.org/en/latest/manual/modules/#module-initialization-and-precompilation if something is missing

mdcfrancis · 2016-01-13T17:33:03Z

@vtjnash as a user error I'd expect the top level package to error on precompile, it does not and causes the crash outlined here. I'd happily see the the precompile error out if all child modules are not so annotated, but I suspect it will break current usage. Perhaps a warning ?

The reproduction supplied boils the issue down to something which does not require a lot of moving parts, but is based on the real problem -

The manifestation of this issue in iJulia etc is due to BinDeps.jl not having precompile set and Conda.jl does (which calls import BinDeps). In isolation this causes no issue (e.g. when you call these on their own), but when you install a package through REQUIRE the 'build step' breaks (this is due to the reuse of the build child process in the package manager.) Specifically you can reproduce the problem by having LibCURL and PyCall in a parent REQUIRE file.

The reason that people see this as intermittent is that it depends on the order of the build and the set of packages specified in the build. Additionally once you have successfully precompiled you are good to go from then on. It is only when you try and build a package with many dependencies in a clean area that this rears its head.

I hope that helps.

tkelman · 2016-01-13T17:44:09Z

If BinDeps is safe to precompile, we could fix that case by adding the precompile directive to it. Better checking, if possible, would be worth trying to do.

mdcfrancis · 2016-01-13T17:53:01Z

It certainly seems safe since any users of Conda are doing this today.

Fixes #13200

JeffBezanson · 2016-02-12T18:09:06Z

I think we should reopen this. If we don't support precompiling a module whose dependencies can't be precompiled, we should throw an error as soon as possible to prevent unpredictable breakage later.

vtjnash · 2016-02-12T18:10:49Z

that's not really something the system can test for

tkelman · 2016-02-12T18:40:33Z

We can't check whether dependencies have marked __precompile__(true) ?

Fixes #13200 (cherry picked from commit 59ca442) ref #14773

sbromberger · 2016-03-25T18:59:44Z

FWIW, I'm running into this today with using Gadfly. Not quite sure how to fix.

Actually, I'm getting the ERROR: LoadError: MethodError: 'module_uuid' has no method matching module_uuid(::Type{ColorTypes.Color{T,N}}) message.

Edit: This was fixed by nuking ~/.julia/lib/0.4 and forcing a rebuild of Gadfly.

skanskan · 2018-01-23T20:46:14Z

I've just installed VS Code and Julia v0.7 and i'm having this same problem:

C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\Nullables\src\nullable.jl:392
in expression starting at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\Nullables\src\Nullables.jl:4
ERROR: LoadError: LoadError: Failed to precompile Nullables to C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\julia_pkgdir\lib\v0.7\Nullables.ji.
Stacktrace:
[1] error at .\error.jl:33 [inlined]
[2] compilecache(::String) at .\loading.jl:648
[3] compilecache at .\loading.jl:605 [inlined]
[4] _require(::Symbol) at .\loading.jl:440
[5] require(::Symbol) at .\loading.jl:333
[6] include(::Module, ::String) at .\boot.jl:292
[7] include_relative(::Module, ::String) at .\loading.jl:521
[8] include at .\sysimg.jl:26 [inlined]
[9] include(::String) at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\JSON\src\JSON.jl:3
[10] top-level scope
[11] include(::Module, ::String) at .\boot.jl:292
[12] include_relative(::Module, ::String) at .\loading.jl:521
[13] include(::Module, ::String) at .\sysimg.jl:26
[14] top-level scope
[15] eval at .\boot.jl:295 [inlined]
[16] top-level scope at .<missing>:2
in expression starting at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\JSON\src\Parser.jl:5
in expression starting at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\JSON\src\JSON.jl:10
ERROR: LoadError: Failed to precompile JSON to C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\julia_pkgdir\lib\v0.7\JSON.ji.
Stacktrace:
[1] error at .\error.jl:33 [inlined]

and so on

yuyichao added the precompile Precompilation of modules label Sep 18, 2015

yuyichao mentioned this issue Sep 18, 2015

[release-0.4] backports for RC2 #13182

Merged

tkelman changed the title ~~Broken precompile on OSX?~~ Intermittently broken precompile Sep 24, 2015

timholy mentioned this issue Sep 24, 2015

Failing to load FixedPointNumbers JuliaMath/FixedPointNumbers.jl#29

Closed

JeffBezanson added the bug Indicates an unexpected problem or unintended behavior label Oct 23, 2015

This was referenced Nov 8, 2015

Trying to update metadata error building IJulia #13915

Closed

Problems installing IJulia with Julia 0.4 on Ubuntu JuliaLang/IJulia.jl#386

Closed

This was referenced Dec 1, 2015

Pkg.build("IJulia") Failing due to precompile fail JuliaLang/IJulia.jl#394

Closed

Julia v0.4, Blink 0.1.5, Gadfly 0.3.14: odd interference of 'using Gadfly' with 'using Blink' JuliaGizmos/Blink.jl#9

Closed

install error 0.4.1 on El Capitan #14213

Closed

yuyichao mentioned this issue Dec 9, 2015

IJulia is broken JuliaLang/IJulia.jl#398

Closed

yuyichao mentioned this issue Jan 13, 2016

Order dependent crash with precompile in 0.4.3 #14663

Closed

yuyichao added a commit that referenced this issue Jan 23, 2016

Document requirement of __precompile__

af16044

Fixes #13200

yuyichao mentioned this issue Jan 23, 2016

Document requirement of __precompile__ #14773

Merged

yuyichao added a commit that referenced this issue Jan 23, 2016

Document requirement of __precompile__

59ca442

Fixes #13200

vtjnash closed this as completed in #14773 Jan 24, 2016

JeffBezanson added the packages Package management and loading label Feb 12, 2016

tkelman pushed a commit that referenced this issue Mar 7, 2016

Document requirement of __precompile__

f5a38dc

Fixes #13200 (cherry picked from commit 59ca442) ref #14773

cstjean mentioned this issue Jun 9, 2016

__precompile__ issues JuliaLang/IJulia.jl#428

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittently broken precompile #13200

Intermittently broken precompile #13200

yuyichao commented Sep 18, 2015

Keno commented Sep 18, 2015

yuyichao commented Sep 19, 2015

yuyichao commented Sep 19, 2015

jiahao commented Sep 19, 2015

tkelman commented Sep 24, 2015

Keno commented Sep 26, 2015

timholy commented Sep 26, 2015

tkelman commented Oct 3, 2015

mdcfrancis commented Jan 13, 2016

vtjnash commented Jan 13, 2016

mdcfrancis commented Jan 13, 2016

tkelman commented Jan 13, 2016

mdcfrancis commented Jan 13, 2016

JeffBezanson commented Feb 12, 2016

vtjnash commented Feb 12, 2016

tkelman commented Feb 12, 2016

sbromberger commented Mar 25, 2016

skanskan commented Jan 23, 2018

Intermittently broken precompile #13200

Intermittently broken precompile #13200

Comments

yuyichao commented Sep 18, 2015

Keno commented Sep 18, 2015

yuyichao commented Sep 19, 2015

yuyichao commented Sep 19, 2015

jiahao commented Sep 19, 2015

tkelman commented Sep 24, 2015

Keno commented Sep 26, 2015

timholy commented Sep 26, 2015

tkelman commented Oct 3, 2015

mdcfrancis commented Jan 13, 2016

vtjnash commented Jan 13, 2016

mdcfrancis commented Jan 13, 2016

tkelman commented Jan 13, 2016

mdcfrancis commented Jan 13, 2016

JeffBezanson commented Feb 12, 2016

vtjnash commented Feb 12, 2016

tkelman commented Feb 12, 2016

sbromberger commented Mar 25, 2016

skanskan commented Jan 23, 2018