Lock finalizers' lists at exit

[kpamnany]

We have occasionally seen memory corruption errors at the end of runs, such as:

Stack trace (from 2 threads, mixed together).

double free or corruption (!prev)

signal (6): Aborted
in expression starting at none:0
error in running finalizer: MethodError(f=Base.AsyncCondition(handle=0x00000000008feb10, cond=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=Task(next=nothing, queue=<circular reference @-2>, storage=nothing, donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=Task(next=nothing, queue=<circular reference @-2>, storage=nothing, donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=nothing, tail=nothing), lock=Base.Threads.SpinLock(owned=0)), result=nothing, logstate=nothing, code=Base.var"#611", rngState0=0xdf5f0d8bd23d416b, rngState1=0x0a255ccbafb1a1fa, rngState2=0x3fd62529a488eea1, rngState3=0x05c091e535deb2c4, _state=0x00, sticky=false, _isexception=false, priority=0x0000), tail=Task(next=nothing, queue=<circular reference @-2>, storage=nothing, donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=nothing, tail=nothing), lock=Base.Threads.SpinLock(owned=0)), result=nothing, logstate=nothing, code=Base.var"#611", rngState0=0xdf5f0d8bd23d416b, rngState1=0x0a255ccbafb1a1fa, rngState2=0x3fd62529a488eea1, rngState3=0x05c091e535deb2c4, _state=0x00, sticky=false, _isexception=false, priority=0x0000)), lock=Base.Threads.SpinLock(owned=0)), result=nothing, logstate=nothing, code=Profile.var"#3", rngState0=0xd837ea2798675862, rngState1=0x6bd6b44513577585, rngState2=0xc8b0102cce0c51ce, rngState3=0x99df76eb80250e05, _state=0x00, sticky=false, _isexception=false, priority=0x0000), tail=Task(next=nothing, queue=<circular reference @-2>, storage=nothing, donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=Task(next=nothing, queue=<circular reference @-2>, storage=nothing, donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=nothing, tail=nothing), lock=Base.Threads.SpinLock(owned=0)), result=nothing, logstate=nothing, code=Base.var"#611", rngState0=0xdf5f0d8bd23d416b, rngState1=0x0a255ccbafb1a1fa, rngState2=0x3fd62529a488eea1, rngState3=0x05c091e535deb2c4, _state=0x00, sticky=false, _isexception=false, priority=0x0000), tail=Task(next=nothing, queue=<circular reference @-2>, storage=nothing, donenotify=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=nothing, tail=nothing), lock=Base.Threads.SpinLock(owned=0)), result=nothing, logstate=nothing, code=Base.var"#611", rngState0=0xdf5f0d8bd23d416b, rngState1=0x0a255ccbafb1a1fa, rngState2=0x3fd62529a488eea1, rngState3=0x05c091e535deb2c4, _state=0x00, sticky=false, _isexception=false, priority=0x0000)), lock=Base.Threads.SpinLock(owned=0)), result=nothing, logstate=nothing, code=Profile.var"#3", rngState0=0xd837ea2798675862, rngState1=0x6bd6b44513577585, rngState2=0xc8b0102cce0c51ce, rngState3=0x99df76eb80250e05, _state=0x00, sticky=false, _isexception=false, priority=0x0000)), lock=Base.Threads.SpinLock(owned=0)), isopen=true, set=false), args=(Base.uvfinalize,), world=0x0000000000007dfc)
gsignal at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
abort at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
error in running finalizer: MethodError(f=Base.Timer(handle=0x0000000000000000, cond=Base.GenericCondition{Base.Threads.SpinLock}(waitq=Base.IntrusiveLinkedList{Task}(head=nothing, tail=nothing), lock=Base.Threads.SpinLock(owned=0)), isopen=false, set=false), args=(Base.uvfinalize,), world=0x0000000000007dfc)
jl_method_error_bare at /build/source/src/gf.c:1879
jl_method_error at /build/source/src/gf.c:1897
jl_lookup_generic_ at /build/source/src/gf.c:2530 [inlined]
ijl_apply_generic at /build/source/src/gf.c:2545
jl_method_error_bare at /build/source/src/gf.c:1879
jl_method_error at /build/source/src/gf.c:1897
jl_lookup_generic_ at /build/source/src/gf.c:2530 [inlined]
ijl_apply_generic at /build/source/src/gf.c:2545
jl_apply at /build/source/src/julia.h:1842 [inlined]
run_finalizer at /build/source/src/gc.c:283
jl_gc_run_finalizers_in_list at /build/source/src/gc.c:370
run_finalizers at /build/source/src/gc.c:413
__libc_message at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
malloc_printerr at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
jl_gc_run_pending_finalizers at /build/source/src/gc.c:426
_int_free at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
_int_realloc at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
realloc at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
jl_apply at /build/source/src/julia.h:1842 [inlined]
run_finalizer at /build/source/src/gc.c:283
arraylist_grow at /build/source/src/support/arraylist.c:58 [inlined]
jl_gc_run_finalizers_in_list at /build/source/src/gc.c:370
arraylist_push at /build/source/src/support/arraylist.c:69
jl_gc_run_finalizers_in_list at /build/source/src/gc.c:362
run_finalizers at /build/source/src/gc.c:413
run_finalizers at /build/source/src/gc.c:413
jl_gc_run_pending_finalizers at /build/source/src/gc.c:426
jl_gc_run_pending_finalizers at /build/source/src/gc.c:426
enable_finalizers at ./gcutils.jl:121 [inlined]
unlock at ./locks-mt.jl:66 [inlined]
unlock at ./locks-mt.jl:66 [inlined]
trylock at ./locks-mt.jl:57 [inlin
]
trylock at ./locks-mt.jl:57 [inlin
multiq_deletemin at ./partr.jl:153

We found that jl_atexit_hook calls jl_gc_run_all_finalizers calls schedule_all_finalizers calls schedule_finalization which pushes to the to_finalize global list without locking. This list is mutated by run_finalizers which can run when a lock is released, as shown in the stack trace above.

The use of finalizers_lock is pretty messy -- on one path it is locked in one function and unlocked in another, while on another path it is locked and unlocked in the same function... all this could use a clean up, but for now, I've implemented a minimal fix in this PR.

This fix seems to have eliminated the crash we've been seeing but we're still running more tests.

[NHDaly]

👍 awesome, thanks. 👍

+1 that it'd be even better to clean this up by making the lock much tighter, only directly around the accesses, but that can come later.

[kpamnany]

@KristofferC: backport to 1.9.1?

[NHDaly]

We backported #49868, and so we should do this one too, which is the other half of the same bug. I added the label.