Unreachable reached when computing the Jacobian with Zygote if an other package is imported before, on Julia < 1.6

[sylvaticus]

I am experiencing a strange Julia crash on Julia < 1.6 when I import BetaML before computing a gradient operation with Zygote, but if I first do this operation, then I import BetaML (or using BetaML) and then do again the gradient operation with Zygote works fine.

Note that in Julia 1.6 it works in all cases.

# --------------------------------------------------------------------------
# OK 
using Statistics, Zygote;
function autoJacobianDifferentName(f,x;nY=length(f(x)))
    x = convert(Array{Float64,1},x)
    j = Array{Float64, 2}(undef, nY, size(x,1))
    for i in 1:nY
        j[i,:] = gradient(x -> f(x)[i], x)[1]'
    end
    return j
end;
pool1dDifferentName(x,poolSize=3;f=mean) = [f(x[i:i+poolSize-1]) for i in 1:length(x)-poolSize+1];
f = (x->pool1dDifferentName(x,2,f=mean));
v = f([1,2,3,4,5]);
J = autoJacobianDifferentName(f,[1,2,3,4,5])
import BetaML
J = autoJacobianDifferentName(f,[1,2,3,4,5])

# --------------------------------------------------------------------------
# Crash in Julia < 1.6, but ok in Julia 1.6

using Statistics, Zygote;
function autoJacobianDifferentName(f,x;nY=length(f(x)))
    x = convert(Array{Float64,1},x)
    j = Array{Float64, 2}(undef, nY, size(x,1))
    for i in 1:nY
        j[i,:] = gradient(x -> f(x)[i], x)[1]'
    end
    return j
end;
pool1dDifferentName(x,poolSize=3;f=mean) = [f(x[i:i+poolSize-1]) for i in 1:length(x)-poolSize+1] ;
f = (x->pool1dDifferentName(x,2,f=mean));
v = f([1,2,3,4,5]);
import BetaML
J = autoJacobianDifferentName(f,[1,2,3,4,5])

This is the complete stack:

lobianco@NCY-BETA-ALOBIANCO:~/.julia/dev/BetaML$ julia1.5
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.5) pkg> activate --temp
 Activating new environment at `/tmp/jl_lsGMKp/Project.toml`

(jl_lsGMKp) pkg> add Zygote, BetaML
   Updating registry at `~/.julia/registries/General`
######################################################################## 100,0%
  Resolving package versions...
Updating `/tmp/jl_lsGMKp/Project.toml`
  [024491cd] + BetaML v0.4.1
  [e88e6eb3] + Zygote v0.6.8
Updating `/tmp/jl_lsGMKp/Manifest.toml`
  [621f4979] + AbstractFFTs v1.0.1
  [56f22d72] + Artifacts v1.3.0
  [024491cd] + BetaML v0.4.1
  [324d7699] + CategoricalArrays v0.9.5
  [082447d4] + ChainRules v0.7.59
  [d360d2e6] + ChainRulesCore v0.9.37
  [861a8166] + Combinatorics v1.0.2
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.26.0
  [e66e0078] + CompilerSupportLibraries_jll v0.3.4+0
  [9a962f9c] + DataAPI v1.6.0
  [864edb3b] + DataStructures v0.18.9
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.0.2
  [31c24e10] + Distributions v0.24.15
  [1a297f60] + FillArrays v0.11.7
  [f6369f11] + ForwardDiff v0.10.18
  [7869d1d1] + IRTools v0.4.2
  [692b3bcd] + JLLWrappers v1.2.0
  [682c06a0] + JSON v0.21.1
  [e80e1ace] + MLJModelInterface v0.4.1
  [1914dd2f] + MacroTools v0.5.6
  [e1d29d7a] + Missings v0.4.5
  [77ba4419] + NaNMath v0.3.5
  [efe28fd5] + OpenSpecFun_jll v0.5.3+4
  [bac558e1] + OrderedCollections v1.4.0
  [90014a1f] + PDMats v0.11.0
  [69de0a69] + Parsers v1.1.0
  [92933f4c] + ProgressMeter v1.5.0
  [1fd47b50] + QuadGK v2.4.1
  [189a3867] + Reexport v1.0.0
  [ae029012] + Requires v1.1.3
  [79098fc4] + Rmath v0.6.1
  [f50d1b31] + Rmath_jll v0.2.2+2
  [321657f4] + ScientificTypes v1.1.1
  [a2af1166] + SortingAlgorithms v0.3.1
  [276daf66] + SpecialFunctions v1.3.0
  [90137ffa] + StaticArrays v1.1.0
  [64bff920] + StatisticalTraits v0.1.1
  [2913bbd2] + StatsBase v0.33.5
  [4c63d2b9] + StatsFuns v0.9.7
  [856f2bd8] + StructTypes v1.5.2
  [e88e6eb3] + Zygote v0.6.8
  [700de1a5] + ZygoteRules v0.2.1
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> using Statistics, Zygote;
[ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

julia> function autoJacobianDifferentName(f,x;nY=length(f(x)))
           x = convert(Array{Float64,1},x)
           j = Array{Float64, 2}(undef, nY, size(x,1))
           for i in 1:nY
               j[i,:] = gradient(x -> f(x)[i], x)[1]'
           end
           return j
       end;

julia> pool1dDifferentName(x,poolSize=3;f=mean) = [f(x[i:i+poolSize-1]) for i in 1:length(x)-poolSize+1] ;

julia> f = (x->pool1dDifferentName(x,2,f=mean));

julia> v = f([1,2,3,4,5]);

julia> import BetaML
[ Info: Precompiling BetaML [024491cd-cc6b-443e-8034-08ea7eb7db2b]

julia> J = autoJacobianDifferentName(f,[1,2,3,4,5])
Unreachable reached at 0x7fdbb0920d30

signal (4): Illegal instruction
in expression starting at REPL[9]:1
prune! at /home/lobianco/.julia/packages/IRTools/aSVI5/src/passes/passes.jl:189
|> at ./operators.jl:834 [inlined]
#IR#15 at /home/lobianco/.julia/packages/IRTools/aSVI5/src/ir/wrap.jl:146
IR at /home/lobianco/.julia/packages/IRTools/aSVI5/src/ir/wrap.jl:142 [inlined]
_lookup_grad at /home/lobianco/.julia/packages/Zygote/lwmfx/src/compiler/emit.jl:101
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
#s3004#1176 at /home/lobianco/.julia/packages/Zygote/lwmfx/src/compiler/interface2.jl:20 [inlined]
#s3004#1176 at ./none:0
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
GeneratedFunctionStub at ./boot.jl:527
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
jl_call_staged at /buildworker/worker/package_linux64/build/src/method.c:376
jl_code_for_staged at /buildworker/worker/package_linux64/build/src/method.c:423
get_staged at ./compiler/utilities.jl:94
retrieve_code_info at ./compiler/utilities.jl:105 [inlined]
InferenceState at ./compiler/inferencestate.jl:118
typeinf_edge at ./compiler/typeinfer.jl:475
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_ext at ./compiler/typeinfer.jl:570
typeinf_ext at ./compiler/typeinfer.jl:601
jfptr_typeinf_ext_23344.clone_1 at /home/lobianco/lib/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
jl_type_infer at /buildworker/worker/package_linux64/build/src/gf.c:296
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:290
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1964
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1919 [inlined]
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2224 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:117
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:206
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:157 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:566
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:840
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:790
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:883
eval at ./boot.jl:331
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
repl_backend_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
start_repl_backend at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
#run_repl#37 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
run_repl at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
#807 at ./client.jl:399
jfptr_YY.807_45002.clone_1 at /home/lobianco/lib/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:655
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:705
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
run_main_repl at ./client.jl:383
exec_options at ./client.jl:313
_start at ./client.jl:506
jfptr__start_53898.clone_1 at /home/lobianco/lib/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
unknown function (ip: 0x401931)
unknown function (ip: 0x401533)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 76029324 (Pool: 76012494; Big: 16830); GC: 77
Illegal instruction (core dumped)

[DhairyaLGandhi]

Seems like betaml might be doing some piracy?

[sylvaticus]

I see.. but no (I am the author of BetaML), I do not think I do any type piracy there...

[DhairyaLGandhi]

Is there the same issue on 1.6? Or can we also get a reproducible example on 1.6

[sylvaticus]

No, on Julia 1.6 it works "normally". The problem is only in Julia < 1.6 (with same versions of packages).

I have several versions of Julia, so I can start a terminal with julia1.5 or julia1.6, and the problem arise only in Julia < 1.6..

[ToucheSir]

A similar error popped up in https://julialang.zulipchat.com/#narrow/stream/137791-general/topic/Reached.20the.20unreachable, so I wonder if they're related?

[sylvaticus]

hmm.. not sure.. they seems on different parts (e.g. different lines of "interface2.jl"), he is on windows, I am on Linux, and also this is riproducible with the workflow above..