Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fatal errors during sorting tests #916

Closed
maleadt opened this issue May 20, 2021 · 7 comments
Closed

Fatal errors during sorting tests #916

maleadt opened this issue May 20, 2021 · 7 comments
Labels
bug Something isn't working

Comments

@maleadt
Copy link
Member

maleadt commented May 20, 2021

I've been seeing a couple of segfaults and other early worker exits during sorting tests, e.g.:

signal (15): Terminated
in expression starting at CUDA.jl/test/sorting.jl:4
sigprocmask at /usr/lib/x86_64-linux-gnu/libc.so.6 (unknown line)
put_rs_cache at /julia/bin/../lib/julia/libunwind.so.8 (unknown line)
find_reg_state at /julia/bin/../lib/julia/libunwind.so.8 (unknown line)
_ULx86_64_dwarf_step at /julia/bin/../lib/julia/libunwind.so.8 (unknown line)
_ULx86_64_step at /julia/bin/../lib/julia/libunwind.so.8 (unknown line)
jl_unw_step at /src/stackwalk.c:544 [inlined]
jl_unw_stepn at /src/stackwalk.c:100
rec_backtrace at /src/stackwalk.c:213
record_backtrace at /src/task.c:304
jl_throw at /src/task.c:634
Instruction at LLVM/hHQuD/src/core/instructions.jl:23
terminator at LLVM/hHQuD/src/core/basicblock.jl:33 [inlined]
macro expansion at GPUCompiler/mgZHw/src/ptx.jl:229 [inlined]
macro expansion at TimerOutputs/PZq45/src/TimerOutput.jl:226 [inlined]
macro expansion at GPUCompiler/mgZHw/src/ptx.jl:222 [inlined]
macro expansion at TimerOutputs/PZq45/src/TimerOutput.jl:226 [inlined]
hide_unreachable! at GPUCompiler/mgZHw/src/ptx.jl:197
unknown function (ip: 0x7f7dea0a0681)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
function_pass_callback at LLVM/hHQuD/src/pass.jl:74
unknown function (ip: 0x7f7dea092d41)
runOnFunction at /src/llvm-api.cpp:193
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
LLVMRunPassManager at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
macro expansion at LLVM/hHQuD/src/util.jl:85 [inlined]
LLVMRunPassManager at LLVM/hHQuD/lib/libLLVM_h.jl:4705 [inlined]
run! at LLVM/hHQuD/src/passmanager.jl:39 [inlined]
#51 at GPUCompiler/mgZHw/src/irgen.jl:70
ModulePassManager at LLVM/hHQuD/src/passmanager.jl:33
unknown function (ip: 0x7f7de3569c01)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
macro expansion at TimerOutputs/PZq45/src/TimerOutput.jl:226 [inlined]
irgen at GPUCompiler/mgZHw/src/irgen.jl:53
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
macro expansion at GPUCompiler/mgZHw/src/driver.jl:153 [inlined]
macro expansion at TimerOutputs/PZq45/src/TimerOutput.jl:226 [inlined]
macro expansion at GPUCompiler/mgZHw/src/driver.jl:152 [inlined]
#emit_llvm#86 at GPUCompiler/mgZHw/src/utils.jl:62
emit_llvm##kw at GPUCompiler/mgZHw/src/utils.jl:60
unknown function (ip: 0x7f7de3709864)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
emit_llvm##kw at GPUCompiler/mgZHw/src/utils.jl:60
#codegen#82 at GPUCompiler/mgZHw/src/driver.jl:55
codegen##kw at GPUCompiler/mgZHw/src/driver.jl:48 [inlined]
#93 at GPUCompiler/mgZHw/src/driver.jl:268
get! at ./dict.jl:465
unknown function (ip: 0x7f7de3568c13)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
macro expansion at GPUCompiler/mgZHw/src/driver.jl:267 [inlined]
#emit_llvm#86 at GPUCompiler/mgZHw/src/utils.jl:62
emit_llvm at GPUCompiler/mgZHw/src/utils.jl:60
unknown function (ip: 0x7f7dea07c37d)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
macro expansion at TimerOutputs/PZq45/src/TimerOutput.jl:210 [inlined]
macro expansion at CUDA.jl/src/compiler/execution.jl:299 [inlined]
cufunction_compile at TimerOutputs/PZq45/src/TimerOutput.jl:210
check_cache at GPUCompiler/mgZHw/src/cache.jl:31 [inlined]
cached_compilation at CUDA.jl/src/sorting.jl:373 [inlined]
cached_compilation at GPUCompiler/mgZHw/src/cache.jl:0
macro expansion at CUDA.jl/src/compiler/execution.jl:288 [inlined]
#cufunction#231 at TimerOutputs/PZq45/src/TimerOutput.jl:210
cufunction at TimerOutputs/PZq45/src/TimerOutput.jl:203
unknown function (ip: 0x7f7de355ea47)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
macro expansion at CUDA.jl/src/compiler/execution.jl:102 [inlined]
#quicksort!#9 at CUDA.jl/src/sorting.jl:467
quicksort!##kw at CUDA.jl/src/sorting.jl:458
unknown function (ip: 0x7f7de355d3f5)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
#sort!#338 at CUDA.jl/src/sorting.jl:492
sort!##kw at CUDA.jl/src/sorting.jl:488
unknown function (ip: 0x7f7de355c8c5)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
check_sort!#62 at CUDA.jl/test/sorting.jl:201
unknown function (ip: 0x7f7de355c81a)
check_sort!#117##kw at CUDA.jl/test/sorting.jl:200 [inlined]
check_sort!#117##kw at CUDA.jl/test/sorting.jl:200
unknown function (ip: 0x7f7de355c3e6)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
jl_apply at /src/julia.h:1703 [inlined]
do_call at /src/interpreter.c:115
eval_value at /src/interpreter.c:204
eval_stmt_value at /src/interpreter.c:155 [inlined]
eval_body at /src/interpreter.c:562
eval_body at /src/interpreter.c:490
eval_body at /src/interpreter.c:490
eval_body at /src/interpreter.c:490
eval_body at /src/interpreter.c:490
eval_body at /src/interpreter.c:490
jl_interpret_toplevel_thunk at /src/interpreter.c:670
jl_toplevel_eval_flex at /src/toplevel.c:877
jl_toplevel_eval_flex at /src/toplevel.c:825
jl_toplevel_eval_in at /src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1094
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
_include at ./loading.jl:1148
include at ./client.jl:444 [inlined]
#12 at CUDA.jl/test/runtests.jl:86 [inlined]
macro expansion at CUDA.jl/test/setup.jl:63 [inlined]
macro expansion at /usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
macro expansion at CUDA.jl/test/setup.jl:63 [inlined]
macro expansion at CUDA.jl/src/utilities.jl:28 [inlined]
macro expansion at CUDA.jl/src/pool.jl:556 [inlined]
top-level scope at CUDA.jl/test/setup.jl:62
jl_toplevel_eval_flex at /src/toplevel.c:871
jl_toplevel_eval_in at /src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
runtests at CUDA.jl/test/setup.jl:74
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
jl_apply at /src/julia.h:1703 [inlined]
do_apply at /src/builtins.c:670
#106 at /usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278
run_work_thunk at /usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
macro expansion at /usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278 [inlined]
#105 at ./task.jl:411
unknown function (ip: 0x7f7e1fb15edc)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
jl_apply at /src/julia.h:1703 [inlined]
start_task at /src/task.c:839
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 1401069888 (Pool: 1400073824; Big: 996064); GC: 779

signal (11): Segmentation fault
in expression starting at CUDA.jl/test/sorting.jl:4
_ZN4llvm16LiveIntervalCalc9calculateERNS_12LiveIntervalEb at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm13LiveIntervals22computeVirtRegIntervalERNS_12LiveIntervalE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm12RegAllocBase16allocatePhysRegsEv at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN12_GLOBAL__N_18RAGreedy20runOnMachineFunctionERN4llvm15MachineFunctionE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
operator() at /src/jitlayers.cpp:613
_ZN4llvm3orc14IRCompileLayer4emitENS0_29MaterializationResponsibilityENS0_16ThreadSafeModuleE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeENS0_29MaterializationResponsibilityE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession26materializeOnCurrentThreadESt10unique_ptrINS0_19MaterializationUnitESt14default_deleteIS3_EENS0_29MaterializationResponsibilityE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZNSt17_Function_handlerIFvSt10unique_ptrIN4llvm3orc19MaterializationUnitESt14default_deleteIS3_EENS2_29MaterializationResponsibilityEEPS8_E9_M_invokeERKSt9_Any_dataOS6_OS7_ at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17runOutstandingMUsEv at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_EENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_EENSN_IS6_SV_EEEEEE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_EEEENSL_IS5_EENS_6detail12DenseMapPairIS5_SN_EEEEEE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
addModule at /src/jitlayers.cpp:788
jl_add_to_ee at /src/jitlayers.cpp:1067
jl_add_to_ee at /src/jitlayers.cpp:1111
jl_add_to_ee at /src/jitlayers.cpp:1133 [inlined]
_jl_compile_codeinst at /src/jitlayers.cpp:154
jl_generate_fptr at /src/jitlayers.cpp:352
jl_compile_method_internal at /src/gf.c:1970
jl_compile_method_internal at /src/gf.c:2236 [inlined]
_jl_invoke at /src/gf.c:2229 [inlined]
jl_apply_generic at /src/gf.c:2419
_atexit at ./initdefs.jl:343
jfptr__atexit_31138.clone_1 at /julia/lib/julia/sys.so (unknown line)
_jl_invoke at /src/gf.c:2237 [inlined]
jl_apply_generic at /src/gf.c:2419
jl_apply at /src/julia.h:1703 [inlined]
jl_atexit_hook at /src/init.c:223
jl_exit at /src/jl_uv.c:632
jl_exit_thread0_cb at /src/signals-unix.c:353
unknown function (ip: (nil))
Allocations: 1401069888 (Pool: 1400073824; Big: 996064); GC: 779

Worker 2 terminated.
@maleadt maleadt added the bug Something isn't working label May 20, 2021
@maleadt
Copy link
Member Author

maleadt commented May 20, 2021

Hmm, another one:

signal (15): Terminated
in expression starting at CUDA.jl/test/sorting.jl:4
unknown function (ip: 0x7f1a595e7d05)
__libc_malloc at /usr/lib/x86_64-linux-gnu/libc.so.6 (unknown line)
operator new at /workspace/srcdir/gcc-9.1.0/libstdc++-v3/libsupc++/new_op.cc:50
void std::vector<llvm::DomTreeNodeBase<llvm::BasicBlock>*, std::allocator<llvm::DomTreeNodeBase<llvm::BasicBlock>*> >::emplace_back<llvm::DomTreeNodeBase<llvm::BasicBlock>*>(llvm::DomTreeNodeBase<llvm::BasicBlock>*&&) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::DominatorTreeBase<llvm::BasicBlock, false>::createChild(llvm::BasicBlock*, llvm::DomTreeNodeBase<llvm::BasicBlock>*) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::DomTreeBuilder::SemiNCAInfo<llvm::DominatorTreeBase<llvm::BasicBlock, false> >::attachNewSubtree(llvm::DominatorTreeBase<llvm::BasicBlock, false>&, llvm::DomTreeNodeBase<llvm::BasicBlock>*) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::DomTreeBuilder::SemiNCAInfo<llvm::DominatorTreeBase<llvm::BasicBlock, false> >::CalculateFromScratch(llvm::DominatorTreeBase<llvm::BasicBlock, false>&, llvm::DomTreeBuilder::SemiNCAInfo<llvm::DominatorTreeBase<llvm::BasicBlock, false> >::BatchUpdateInfo*) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::DominatorTreeWrapperPass::runOnFunction(llvm::Function&) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::FPPassManager::runOnFunction(llvm::Function&) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::FPPassManager::runOnModule(llvm::Module&) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::legacy::PassManagerImpl::run(llvm::Module&) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
operator() at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:613
llvm::orc::IRCompileLayer::emit(llvm::orc::MaterializationResponsibility, llvm::orc::ThreadSafeModule) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::BasicIRLayerMaterializationUnit::materialize(llvm::orc::MaterializationResponsibility) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::materializeOnCurrentThread(std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
std::_Function_handler<void (std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility), void (*)(std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility)>::_M_invoke(std::_Any_data const&, std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >&&, llvm::orc::MaterializationResponsibility&&) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::runOutstandingMUs() at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::lookup(llvm::orc::LookupKind, std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet, llvm::orc::SymbolState, llvm::unique_function<void (llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > >)>, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::lookup(std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet const&, llvm::orc::LookupKind, llvm::orc::SymbolState, std::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::lookup(std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib*>, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib*>, llvm::StringRef, llvm::orc::SymbolState) at julia/bin/../lib/julia/libLLVM-11jl.so (unknown line)
addModule at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:788
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1067
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1111
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1096
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1133 [inlined]
_jl_compile_codeinst at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:154
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:352
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1970
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:2236 [inlined]
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2229 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
macro expansion at GPUCompiler/mgZHw/src/driver.jl:267 [inlined]
#emit_llvm#86 at GPUCompiler/mgZHw/src/utils.jl:62
emit_llvm at GPUCompiler/mgZHw/src/utils.jl:60
unknown function (ip: 0x7f1a1016d6bd)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
macro expansion at TimerOutputs/PZq45/src/TimerOutput.jl:210 [inlined]
macro expansion at CUDA.jl/src/compiler/execution.jl:299 [inlined]
cufunction_compile at TimerOutputs/PZq45/src/TimerOutput.jl:210
check_cache at GPUCompiler/mgZHw/src/cache.jl:31 [inlined]
cached_compilation at CUDA.jl/src/sorting.jl:373 [inlined]
cached_compilation at GPUCompiler/mgZHw/src/cache.jl:0
macro expansion at CUDA.jl/src/compiler/execution.jl:288 [inlined]
#cufunction#231 at TimerOutputs/PZq45/src/TimerOutput.jl:210
cufunction at TimerOutputs/PZq45/src/TimerOutput.jl:203
unknown function (ip: 0x7f19fc3665a7)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
macro expansion at CUDA.jl/src/compiler/execution.jl:102 [inlined]
#quicksort!#9 at CUDA.jl/src/sorting.jl:467
quicksort!##kw at CUDA.jl/src/sorting.jl:458
unknown function (ip: 0x7f19fc364f55)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#sort!#339 at CUDA.jl/src/sorting.jl:502
sort!##kw at CUDA.jl/src/sorting.jl:498
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
check_sort!#72 at CUDA.jl/test/sorting.jl:201
unknown function (ip: 0x7f19fc364450)
check_sort!#127##kw at CUDA.jl/test/sorting.jl:200 [inlined]
check_sort!#127##kw at CUDA.jl/test/sorting.jl:200
unknown function (ip: 0x7f19fc3640d0)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1094
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1148
include at ./client.jl:444 [inlined]
#12 at CUDA.jl/test/runtests.jl:86 [inlined]
macro expansion at CUDA.jl/test/setup.jl:63 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
macro expansion at CUDA.jl/test/setup.jl:63 [inlined]
macro expansion at CUDA.jl/src/utilities.jl:28 [inlined]
macro expansion at CUDA.jl/src/pool.jl:556 [inlined]
top-level scope at CUDA.jl/test/setup.jl:62
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:871
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
runtests at CUDA.jl/test/setup.jl:74
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
#106 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278 [inlined]
#105 at ./task.jl:411
unknown function (ip: 0x7f1a4426becc)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:839
unknown function (ip: (nil))
unknown function (ip: (nil))
Allocations: 1408996718 (Pool: 1407998574; Big: 998144); GC: 775

@xaellison
Copy link
Contributor

I get the same (or related) failure when I try to run under nsys launch julia-1.6.1/bin/julia. I'm not sure what to make of it

julia> a = rand(Int32, Int(1e6))
julia> c = CuArray(a)
julia> CUDA.@profile sort!(c)

WARNING: CUDA tracing is required for cudaProfilerStart/Stop API support. Turning it on by default.
waiting for capture range to start the collection
Capture range started in the application

signal (11): Segmentation fault
in expression starting at REPL[4]:1
unknown function (ip: 0x7f0b0351564c)
unknown function (ip: 0x7f0b0363c590)
unknown function (ip: 0x7f0b034f3bd8)
unknown function (ip: 0x7f0b034f10ad)
unknown function (ip: 0x7f0b034be5c4)
unknown function (ip: 0x7f0b034c0a9b)
unknown function (ip: 0x7f0b041d7142)
unknown function (ip: 0x7f0b042853e8)
unknown function (ip: 0x7f0b0422f3f2)
unknown function (ip: 0x7f0b042e71bc)
unsafe_cuModuleLoadDataEx at /home/paperspace/.julia/packages/CUDA/3VnCC/lib/cudadrv/libcuda.jl:258
#19 at /home/paperspace/.julia/packages/CUDA/3VnCC/lib/cudadrv/module.jl:47 [inlined]
retry_reclaim at /home/paperspace/.julia/packages/CUDA/3VnCC/src/pool.jl:420
CuModule at /home/paperspace/.julia/packages/CUDA/3VnCC/lib/cudadrv/module.jl:46
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
CuModule at /home/paperspace/.julia/packages/CUDA/3VnCC/lib/cudadrv/module/linker.jl:163
cufunction_link at /home/paperspace/.julia/packages/CUDA/3VnCC/src/compiler/execution.jl:347
unknown function (ip: 0x7f0a70717f65)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
check_cache at /home/paperspace/.julia/packages/GPUCompiler/eJOtJ/src/cache.jl:56 [inlined]
cached_compilation at /home/paperspace/.julia/packages/CUDA/3VnCC/src/sorting.jl:373 [inlined]
cached_compilation at /home/paperspace/.julia/packages/GPUCompiler/eJOtJ/src/cache.jl:0
#cufunction#261 at /home/paperspace/.julia/packages/CUDA/3VnCC/src/compiler/execution.jl:289
cufunction at /home/paperspace/.julia/packages/CUDA/3VnCC/src/compiler/execution.jl:283
unknown function (ip: 0x7f0b2cafe8d7)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
macro expansion at /home/paperspace/.julia/packages/CUDA/3VnCC/src/compiler/execution.jl:102 [inlined]
#quicksort!#9 at /home/paperspace/.julia/packages/CUDA/3VnCC/src/sorting.jl:467
quicksort!##kw at /home/paperspace/.julia/packages/CUDA/3VnCC/src/sorting.jl:458
unknown function (ip: 0x7f0b2caf0015)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#sort!#369 at /home/paperspace/.julia/packages/CUDA/3VnCC/src/sorting.jl:502
unknown function (ip: 0x7f0b2caef251)
sort! at /home/paperspace/.julia/packages/CUDA/3VnCC/src/sorting.jl:498
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:435
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:525
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:139
repl_backend_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:200
start_repl_backend at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:185
#run_repl#42 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:317
run_repl at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:305
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
#874 at ./client.jl:387
jfptr_YY.874_41532.clone_1 at /home/paperspace/julia-1.6.1/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:714
#invokelatest#2 at ./essentials.jl:708 [inlined]
invokelatest at ./essentials.jl:706 [inlined]
run_main_repl at ./client.jl:372
exec_options at ./client.jl:302
_start at ./client.jl:485
jfptr__start_34289.clone_1 at /home/paperspace/julia-1.6.1/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:51
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/paperspace/julia-1.6.1/bin/julia (unknown line)
Allocations: 47574270 (Pool: 47558050; Big: 16220); GC: 50

The target application terminated with signal 11 (SIGSEGV)
Processing events...
Saving temporary "/tmp/nsys-report-2539-d95b-8bad-8f50.qdstrm" file to disk...

Creating final output files...
Processing [==============================================================100%]
Saved report file to "/tmp/nsys-report-2539-d95b-8bad-8f50.qdrep"
Report file moved to "/home/paperspace/report2.qdrep"

@maleadt
Copy link
Member Author

maleadt commented May 29, 2021

No, this is probably a different failure (an error inside of CUDA vs what's likely a corruption that crashes LLVM). Make sure your CUDA driver and Nsight installation are up to date.

@maleadt
Copy link
Member Author

maleadt commented Jun 1, 2021

Works fine here:

julia> CUDA.@profile sort!(c)
[ Info: Running under Nsight Systems, CUDA.@profile will automatically start the profiler

WARNING: CUDA tracing is required for cudaProfilerStart/Stop API support. Turning it on by default.
waiting for capture range to start the collection
Capture range started in the application
Capture range ended in the application
Processing events...
Saving temporary "/tmp/nsys-report-fb6f-61e1-4ccb-0bcd.qdstrm" file to disk...

Creating final output files...
Processing [==============================================================100%]
Saved report file to "/tmp/nsys-report-fb6f-61e1-4ccb-0bcd.qdrep"
Report file moved to "/home/tim/Julia/pkg/CUDA/report16.qdrep"

command ignored
[ Info: Profiling has finished, open the report listed above with `nsight-sys`
1000000-element CuArray{Int32, 1}:
 -2147480975
 -2147476614
 -2147464328
 -2147462531
 -2147453106
 -2147449845
 -2147442709
 -2147442116
 -2147433336
 -2147431609
           ⋮
  2147421928
  2147422568
  2147455135
  2147455793
  2147455852
  2147461908
  2147462453
  2147468744
  2147471305

@maleadt
Copy link
Member Author

maleadt commented Jun 9, 2021

Haven't seen this in a while, probably fixed as part of some sorting test changes?

@maleadt maleadt closed this as completed Jun 9, 2021
@maleadt
Copy link
Member Author

maleadt commented Aug 13, 2021

Saw it again:

      From worker 4:	signal (15): Terminated
      From worker 4:	in expression starting at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/sorting.jl:4
      From worker 4:	hash at ./float.jl:469
      From worker 4:	hash at ./float.jl:484 [inlined]
      From worker 4:	hash at ./hashing.jl:18 [inlined]
      From worker 4:	hashindex at ./dict.jl:169 [inlined]
      From worker 4:	rehash! at ./dict.jl:203
      From worker 4:	_setindex! at ./dict.jl:369 [inlined]
      From worker 4:	setindex! at ./dict.jl:390
      From worker 4:	setindex! at /root/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/DataStructures/nBjdy/src/accumulator.jl:53 [inlined]
      From worker 4:	inc! at /root/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/DataStructures/nBjdy/src/accumulator.jl:75
      From worker 4:	inc! at /root/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/DataStructures/nBjdy/src/accumulator.jl:76 [inlined]
      From worker 4:	counter at /root/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/DataStructures/nBjdy/src/accumulator.jl:30
      From worker 4:	check_equivalence#30 at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/sorting.jl:173
      From worker 4:	check_equivalence at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/sorting.jl:173
      From worker 4:	unknown function (ip: 0x7fa3a46887f6)
      From worker 4:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 4:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 4:	check_sort!#35 at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/sorting.jl:203
      From worker 4:	unknown function (ip: 0x7fa3a49e8175)
      From worker 4:	check_sort! at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/sorting.jl:200 [inlined]
      From worker 4:	check_sort! at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/sorting.jl:200
      From worker 4:	unknown function (ip: 0x7fa3a49e7e79)
      From worker 4:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 4:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 4:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
      From worker 4:	do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
      From worker 4:	eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
      From worker 4:	eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
      From worker 4:	eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
      From worker 4:	eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
      From worker 4:	eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
      From worker 4:	eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
      From worker 4:	eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
      From worker 4:	eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
      From worker 4:	jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
      From worker 4:	jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
      From worker 4:	jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
      From worker 4:	jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
      From worker 4:	eval at ./boot.jl:360 [inlined]
      From worker 4:	include_string at ./loading.jl:1116
      From worker 4:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 4:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 4:	_include at ./loading.jl:1170
      From worker 4:	include at ./client.jl:444 [inlined]
      From worker 4:	#12 at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/runtests.jl:86 [inlined]
      From worker 4:	macro expansion at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/setup.jl:59 [inlined]
      From worker 4:	macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
      From worker 4:	macro expansion at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/setup.jl:59 [inlined]
      From worker 4:	macro expansion at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/src/utilities.jl:28 [inlined]
      From worker 4:	macro expansion at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/src/pool.jl:473 [inlined]
      From worker 4:	top-level scope at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/setup.jl:58
      From worker 4:	jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:871
      From worker 4:	jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
      From worker 4:	eval at ./boot.jl:360 [inlined]
      From worker 4:	runtests at /var/lib/buildkite-agent/builds/rtx4000-gpuci4-julia-csail-mit-edu/julialang/cuda-dot-jl/test/setup.jl:70
      From worker 4:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 4:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 4:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
      From worker 4:	do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
      From worker 4:	#106 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278
      From worker 4:	run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
      From worker 4:	macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278 [inlined]
      From worker 4:	#105 at ./task.jl:411
      From worker 4:	unknown function (ip: 0x7fa3f052cf0c)
      From worker 4:	_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
      From worker 4:	jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
      From worker 4:	jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
      From worker 4:	start_task at /buildworker/worker/package_linux64/build/src/task.c:839
      From worker 4:	unknown function (ip: (nil))
      From worker 4:	unknown function (ip: (nil))
      From worker 4:	Allocations: 1947320704 (Pool: 1945890251; Big: 1430453); GC: 949

@maleadt maleadt reopened this Aug 13, 2021
@maleadt
Copy link
Member Author

maleadt commented Dec 10, 2021

Again, haven't seen this in a while anymore.

@maleadt maleadt closed this as completed Dec 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants