Julia 1.8 `CVODE_BDF` causes segfaults

[bolognam]

I'm consistently getting segfaults in both Windows 10 and Ubuntu 20.04 on Julia 1.8.0 when running my ODE solves compared to Julia 1.7.2. I've tried the latest in Sundials (v4.10.0) as well as the version I was using in Julia 1.7.2 (v4.9.4).

Here's the stack trace in Windows (Sundials v4.9.4):

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0xa2d095f9 -- N_VLinearSum at C:\Users\Michael\.julia\artifacts\4ccc575631c856942ae91cbb8294de9b0a746c9d\bin\libsundials_cvodes.dll 
(unknown line)
in expression starting at C:\Users\Michael\MyPackage\mycode.jl:30
N_VLinearSum at C:\Users\Michael\.julia\artifacts\4ccc575631c856942ae91cbb8294de9b0a746c9d\bin\libsundials_cvodes.dll (unknown line)
cvNlsResidual at C:\Users\Michael\.julia\artifacts\4ccc575631c856942ae91cbb8294de9b0a746c9d\bin\libsundials_cvodes.dll (unknown line)
SUNNonlinSolSolve_Newton at C:\Users\Michael\.julia\artifacts\4ccc575631c856942ae91cbb8294de9b0a746c9d\bin\libsundials_sunnonlinsolnewton.dll (unknown line)
cvStep at C:\Users\Michael\.julia\artifacts\4ccc575631c856942ae91cbb8294de9b0a746c9d\bin\libsundials_cvodes.dll (unknown line)
CVode at C:\Users\Michael\.julia\artifacts\4ccc575631c856942ae91cbb8294de9b0a746c9d\bin\libsundials_cvodes.dll (unknown line)
CVode at C:\Users\Michael\.julia\packages\Sundials\k9hc3\lib\libsundials_api.jl:1523 [inlined]
CVode at C:\Users\Michael\.julia\packages\Sundials\k9hc3\lib\libsundials_api.jl:1528
solver_step at C:\Users\Michael\.julia\packages\Sundials\k9hc3\src\common_interface\solve.jl:1478
#solve!#115 at C:\Users\Michael\.julia\packages\Sundials\k9hc3\src\common_interface\solve.jl:1573
solve! at C:\Users\Michael\.julia\packages\Sundials\k9hc3\src\common_interface\solve.jl:1558
unknown function (ip: 0000000063e290e1)
top-level scope at .\timing.jl:263
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:897
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:850
ijl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:915 [inlined]
ijl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:965
eval at .\boot.jl:368 [inlined]
include_string at .\loading.jl:1428
_include at .\loading.jl:1488
include at .\client.jl:476
unknown function (ip: 000000005b23e1e6)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
do_call at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:126
eval_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:215
eval_stmt_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:166 [inlined]
eval_body at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:594
jl_interpret_toplevel_thunk at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:750
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:906
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:850
ijl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:915 [inlined]
ijl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:965
eval at .\boot.jl:368 [inlined]
eval_user_input at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\REPL\src\REPL.jl:151
repl_backend_loop at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\REPL\src\REPL.jl:247
start_repl_backend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\REPL\src\REPL.jl:232
#run_repl#47 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\REPL\src\REPL.jl:369
run_repl at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.8\REPL\src\REPL.jl:355
jfptr_run_repl_69828.clone_1 at C:\Users\Michael\AppData\Local\Programs\Julia-1.8.0\lib\julia\sys.dll (unknown line)
#966 at .\client.jl:419
jfptr_YY.966_58133.clone_1 at C:\Users\Michael\AppData\Local\Programs\Julia-1.8.0\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:774
#invokelatest#2 at .\essentials.jl:729 [inlined]
invokelatest at .\essentials.jl:726 [inlined]
run_main_repl at .\client.jl:404
exec_options at .\client.jl:318
_start at .\client.jl:522
jfptr__start_57488.clone_1 at C:\Users\Michael\AppData\Local\Programs\Julia-1.8.0\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1838 [inlined]
true_main at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:575
jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:719
mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:59
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 347048866 (Pool: 346978877; Big: 69989); GC: 141

And in Linux (Sundials v4.10.0):

signal (11): Segmentation fault
in expression starting at /home/runner/work/MyPackage/mycode.jl:30
unknown function (ip: 0x77a5580)
Allocations: 540922622 (Pool: 540699010; Big: 223612); GC: 230
ERROR: LoadError: Package MyPackage errored during testing (received signal: 11)
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /opt/hostedtoolcache/julia/1.8.0/x64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
   @ Pkg.Operations /opt/hostedtoolcache/julia/1.8.0/x64/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1813
 [3] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Vector{String}, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::Base.Pairs{Symbol, IOContext{Base.PipeEndpoint}, Tuple{Symbol}, NamedTuple{(:io,), Tuple{IOContext{Base.PipeEndpoint}}}})
   @ Pkg.API /opt/hostedtoolcache/julia/1.8.0/x64/share/julia/stdlib/v1.8/Pkg/src/API.jl:431
 [4] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{Base.PipeEndpoint}, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:coverage, :julia_args, :force_latest_compatible_version), Tuple{Bool, Vector{String}, Bool}}})
   @ Pkg.API /opt/hostedtoolcache/julia/1.8.0/x64/share/julia/stdlib/v1.8/Pkg/src/API.jl:156
 [5] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:coverage, :julia_args, :force_latest_compatible_version), Tuple{Bool, Vector{String}, Bool}}})
   @ Pkg.API /opt/hostedtoolcache/julia/1.8.0/x64/share/julia/stdlib/v1.8/Pkg/src/API.jl:171
 [6] top-level scope
   @ ~/work/_actions/julia-actions/julia-runtest/v1/test_harness.jl:15
 [7] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [8] top-level scope
   @ none:1
in expression starting at /home/runner/work/_actions/julia-actions/julia-runtest/v1/test_harness.jl:7

This is closed source software, but if an NDA is signed then I can provide an rr trace.

[ChrisRackauckas]

The preconditioners test is also doing weird things on v1.8 (#366) and we haven't figured out why. That might be a lead, but it's hard to say in advance.

[bolognam]

This code doesn't have preconditioners, the only options used (all of which error), are max_order=1 or no options.

I've got a less stiff problem working with CVODE_BDF(), maybe that helps?

[bolognam]

Looks like by building Sundials from src (with debug build) and hacking Sundials.jl to use the compiled libraries (instead of Sundials_jll) we can avoid the segfaults. This was built from Sundials 5.2.0 (see tag in git). Unfortunately it seems like the debug build causes the memory space to be different than that of the release build.

[ChrisRackauckas]

@giordano or @staticfloat could we get some help retriggering the build?

[giordano]

You want a debug build of Sundials (presumably v5.2.0)? As the official build or something to be used only for testing?

[ChrisRackauckas]

I don't think it needs to be the debug build? But I'm also confused why there's now a memory issue with the old builds.

[bolognam]

So the current jll that's used with Sundials.jl is 5.2.1+0, which presumably is very close the C++ Sundials v5.2.0 code, is what I compiled from source. The release build of C++ Sundials code causes the segfaults in Julia 1.8, but when I switch to the debug build of C++ Sundials code I don't get segfaults in Julia 1.8.

Remember, Julia 1.7 works just fine with the Sundials jll, but for Julia 1.8 I get segfaults.

[giordano]

Well, in Julia v1.7 and v1.8 you should get the same binary. I don't see how rebuilding would help anything here. Maybe the problem is in how the routines from the libraries are called.

[bolognam]

My guess is that a function pointer got garbage collected (i.e. the unknown function message in Linux) and that's how we're segfaulting. I understand that we should get the same Sundials binary between Julia 1.7 and 1.8, so I'm guessing that the way Julia is handling memory has changed.

Using a debug build of C++ Sundials would change how memory is managed overall in the computer, which is why re-building Sundials with a debug build removes the segfault issue I experienced.

[ChrisRackauckas]

It would be helpful for someone to track down what exactly is getting garbage collected too early here.

[bolognam]

If someone can guide me through the process that would help me. Would I build julia from source with assertions to debug this?

[sjdaines]

I just retested our application (that uses Kinsol), putting GC.enable(false) and GC.enable(true) around the call to Sundials, and this appears to remove the segfault (an example failure was in PALEOtoolkit/PALEOmodel.jl#25). So this is at least consistent with the suggestion above that the problem is connected to something getting garbage collected in Julia 1.8 (NB: not a definitive test as the failures are not fully reproducable, so this could just be a statistical fluke).

Some speculation on a possible cause: looking at the Sundials.jl code, it seems to me that there might be a problem here with the use of convert by the NVector wrapper, which returns an N_Vector for use by the Sundials C API, but as I understand it can't be safely used by ccall in this way as it there is no guarantee about the lifetime of the nv.ref_nv::Ref{N_Vector} object:

Sundials.jl/src/nvector_wrapper.jl

Line 61 in c7469aa

Base.convert(::Type{N_Vector}, nv::NVector) = nv.ref_nv[]

The NVector and hence the nv.ref_nv::Ref{N_Vector} may well be temporary, in which case can the garbage collector delete the ref_nv and call the finalizer (which calls N_VDestroy_Serial(ref_nv[]) to free the C memory), before the ccall uses the N_Vector C pointer?

The most obvious case (which explicitly creates temporary NVector objects) is:

Sundials.jl/src/nvector_wrapper.jl

Lines 65 to 73 in c7469aa

	""" `N_Vector(v::Vector{T})`
	Converts Julia `Vector` to `N_Vector`.
	Implicitly creates `NVector` object that manages automatic
	destruction of `N_Vector` object when no longer in use.
	"""
	Base.convert(::Type{N_Vector}, v::Vector{realtype}) = N_Vector(NVector(v))
	Base.convert(::Type{N_Vector}, v::Vector{T}) where {T <: Real} = N_Vector(NVector(v))

but in fact this pattern is used widely by autogenerated code, eg

Sundials.jl/lib/libsundials_api.jl

Lines 1840 to 1849 in c7469aa

	function CVode(cvode_mem, tout::realtype, yout::N_Vector, tret, itask::Cint)
	ccall((:CVode, libsundials_cvodes), Cint,
	(CVODEMemPtr, realtype, N_Vector, Ptr{realtype}, Cint), cvode_mem, tout, yout,
	tret, itask)
	end
	function CVode(cvode_mem, tout, yout, tret, itask)
	__yout = convert(NVector, yout)
	CVode(cvode_mem, tout, convert(N_Vector, __yout), tret, convert(Cint, itask))
	end

where the problem in this case could be that __yout.ref_nv is garbage collected after convert(N_Vector, __yout) but before the ccall ?

I admit I can't understand the documentation for unsafe_convert, https://docs.julialang.org/en/v1/base/c/#Base.unsafe_convert, but looking at the implementation of ccall
https://github.com/JuliaLang/julia/blob/afb6c60d69a38e8a2442a0c7e87c47b8880ad294/base/c.jl#L663-L671
it looks like this is the problem that a combination of cconvert and unsafe_convert is intended to solve ? (see also https://discourse.julialang.org/t/how-to-keep-a-reference-for-c-structure-to-avoid-gc/9310/5).

If this is the problem, then given that there is a comment contradicting this in generate.jl:

Sundials.jl/gen/generate.jl

Lines 134 to 138 in c7469aa

	if arg_type_expr == :N_Vector
	# first convert argument to NVector() and store in local var,
	# this guarantees that the wrapper and associated Sundials object (e.g. N_Vector)
	# is not removed by GC
	return (arg_name_expr,

and if this comment is in fact incorrect there could I imagine be other similar problems, any fix will need to touch (or at least regenerate) a lot of code. So perhaps the first priority is to understand the intended usage of ccall, unsafe_convert, and cconvert and review the design here?

[ChrisRackauckas]

Note that the required vectors are stored in the Julia types for the function and integrator:

https://github.com/SciML/Sundials.jl/blob/master/src/common_interface/integrator_types.jl#L63
https://github.com/SciML/Sundials.jl/blob/master/src/common_interface/function_types.jl#L11-L12
https://github.com/SciML/Sundials.jl/blob/master/src/common_interface/integrator_utils.jl#L140

So in theory the GC would know those finalizers are not hit until after the integrator's reference is gone. At least, that's how it was working before v1.8.