Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nightly + 1.10-alpha1: GC marking segfault during GAP tests #2543

Closed
benlorenz opened this issue Jul 11, 2023 · 12 comments
Closed

nightly + 1.10-alpha1: GC marking segfault during GAP tests #2543

benlorenz opened this issue Jul 11, 2023 · 12 comments
Labels
bug Something isn't working nightly error with julia nightly or pre-releases package: GAP

Comments

@benlorenz
Copy link
Member

benlorenz commented Jul 11, 2023

Another rare crash, this time during the GAP OscarInterface tests, log:

-> Testing GAP/oscar_to_gap.jl took: runtime 0.082 seconds + compilation 2.391 seconds + recompilation 0.000 seconds, 89.49 MB
Architecture: x86_64-pc-linux-gnu-julia1.11-64-kv8

testing: /home/runner/work/Oscar.jl/Oscar.jl/src/../gap/OscarInterface//tst/al\
nuth/ALNUTH.tst
   20958 ms (1274 ms GC) and 5.58MB allocated for alnuth/ALNUTH.tst
testing: /home/runner/work/Oscar.jl/Oscar.jl/src/../gap/OscarInterface//tst/al\
nuth/examples.tst
    1444 ms (915 ms GC) and 15.2MB allocated for alnuth/examples.tst
testing: /home/runner/work/Oscar.jl/Oscar.jl/src/../gap/OscarInterface//tst/al\
nuth/manual.tst
    1319 ms (1096 ms GC) and 2.90MB allocated for alnuth/manual.tst
testing: /home/runner/work/Oscar.jl/Oscar.jl/src/../gap/OscarInterface//tst/al\
nuth/polynome.tst

[3037] signal (11.1): Segmentation fault
in expression starting at /home/runner/work/Oscar.jl/Oscar.jl/test/GAP/runtests.jl:11
gc_mark_outrefs at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:2499 [inlined]
gc_mark_loop_serial_ at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:2669
gc_mark_loop_serial at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:2692
gc_mark_loop at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:2805 [inlined]
_jl_gc_collect at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:3124
ijl_gc_collect at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:3446
maybe_collect at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:916 [inlined]
ijl_gc_counted_malloc at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:3610
ijl_malloc at /cache/build/default-amdci4-1/julialang/julia-master/src/gc.c:3672
flint_malloc at /workspace/srcdir/flint2/memory_manager.c:112
_fmpz_poly_mullow_SS at /workspace/srcdir/flint2/fmpz_poly/mullow_SS.c:87
_fq_nmod_poly_mullow_KS at /workspace/srcdir/flint2/fq_poly_templates/mullow_KS.c:59
fq_nmod_poly_mullow at /workspace/srcdir/flint2/fq_poly_templates/mullow.c:68
* at /home/runner/.julia/packages/Nemo/ObBjV/src/flint/fq_abs_series.jl:245
polynomial_to_power_sums at /home/runner/.julia/packages/AbstractAlgebra/lDlYU/src/Poly.jl:2900
norm_mod at /home/runner/.julia/packages/Hecke/PJiPq/src/NumField/NfAbs/PolyFact.jl:887
macro expansion at /home/runner/.julia/packages/Hecke/PJiPq/src/Hecke.jl:500 [inlined]
factor_trager at /home/runner/.julia/packages/Hecke/PJiPq/src/NumField/NfAbs/Elem.jl:376
#_factor#1522 at /home/runner/.julia/packages/Hecke/PJiPq/src/NumField/NfAbs/Elem.jl:354
_factor at /home/runner/.julia/packages/Hecke/PJiPq/src/NumField/NfAbs/Elem.jl:348 [inlined]
#factor#1521 at /home/runner/.julia/packages/Hecke/PJiPq/src/NumField/NfAbs/Elem.jl:335
factor at /home/runner/.julia/packages/Hecke/PJiPq/src/NumField/NfAbs/Elem.jl:309
unknown function (ip: 0x7fa0b317edd5)
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
jl_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/julia.h:1966 [inlined]
do_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/builtins.c:768
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
jl_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/julia.h:1966 [inlined]
do_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/builtins.c:768
_apply at ./boot.jl:836
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
jl_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/julia.h:1966 [inlined]
ijl_call2 at /cache/build/default-amdci4-1/julialang/julia-master/src/jlapi.c:258
DoCallJuliaFunc at /workspace/srcdir/GAP.jl/pkg/JuliaInterface/src/calls.c:113 [inlined]
DoCallJuliaFunc2Arg at /workspace/srcdir/GAP.jl/pkg/JuliaInterface/src/calls.c:147
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:150 [inlined]
EvalFunccall2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:308
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecReturnObj at /workspace/srcdir/gap-4.12.2/src/stats.c:965
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat2 at /workspace/srcdir/gap-4.12.2/src/stats.c:165
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:[468](https://github.com/oscar-system/Oscar.jl/actions/runs/5516967428/jobs/10058976212?pr=2542#step:6:471) [inlined]
DoExecFunc2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:495
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
CallNArgs<2> at /workspace/srcdir/gap-4.12.2/src/opers.cc:1829 [inlined]
DoOperationNArgs<2, 0, 0> at /workspace/srcdir/gap-4.12.2/src/opers.cc:1986 [inlined]
DoOperation2Args at /workspace/srcdir/gap-4.12.2/src/opers.cc:2005
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:139 [inlined]
EvalFunccall1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:303
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecAssLVar at /workspace/srcdir/gap-4.12.2/src/vars.c:167
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:495
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:150 [inlined]
EvalFunccall2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:308
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecForHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:372 [inlined]
ExecFor3 at /workspace/srcdir/gap-4.12.2/src/stats.c:468
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:489
CALL_1ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:311 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:147 [inlined]
EvalFunccall1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:303
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecAssLVar at /workspace/srcdir/gap-4.12.2/src/vars.c:167
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:495
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:150 [inlined]
EvalFunccall2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:308
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecAssLVar at /workspace/srcdir/gap-4.12.2/src/vars.c:167
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat at /workspace/srcdir/gap-4.12.2/src/stats.c:160
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecWhileHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:674 [inlined]
ExecWhile at /workspace/srcdir/gap-4.12.2/src/stats.c:688
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:495
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:150 [inlined]
EvalFunccall2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:308
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecAssLVar at /workspace/srcdir/gap-4.12.2/src/vars.c:167
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat5 at /workspace/srcdir/gap-4.12.2/src/stats.c:180
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:489
CALL_1ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:311 [inlined]
CallNArgs<1> at /workspace/srcdir/gap-4.12.2/src/opers.cc:1826 [inlined]
DoOperationNArgs<1, 0, 0> at /workspace/srcdir/gap-4.12.2/src/opers.cc:1986 [inlined]
DoOperation1Args at /workspace/srcdir/gap-4.12.2/src/opers.cc:2000
DoAttribute at /workspace/srcdir/gap-4.12.2/src/opers.cc:2332
CALL_1ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:311 [inlined]
IntrFuncCallEnd at /workspace/srcdir/gap-4.12.2/src/intrprtr.c:397
EvalRef at /workspace/srcdir/gap-4.12.2/src/read.c:382
ReadCallVarAss at /workspace/srcdir/gap-4.12.2/src/read.c:889
ReadAtom at /workspace/srcdir/gap-4.12.2/src/read.c:1603
ReadFactor at /workspace/srcdir/gap-4.12.2/src/read.c:1674
ReadTerm at /workspace/srcdir/gap-4.12.2/src/read.c:1723
ReadAri at /workspace/srcdir/gap-4.12.2/src/read.c:1755
ReadRel at /workspace/srcdir/gap-4.12.2/src/read.c:1792
ReadAnd at /workspace/srcdir/gap-4.12.2/src/read.c:1829
ReadExpr at /workspace/srcdir/gap-4.12.2/src/read.c:1904
ReadEvalCommand at /workspace/srcdir/gap-4.12.2/src/read.c:2577
FuncREAD_STREAM_LOOP at /workspace/srcdir/gap-4.12.2/src/streams.c:895
CALL_3ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:321 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:153 [inlined]
ExecProccall3args at /workspace/srcdir/gap-4.12.2/src/funcs.c:230
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat at /workspace/srcdir/gap-4.12.2/src/stats.c:160
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecForRangeHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:538 [inlined]
ExecForRange at /workspace/srcdir/gap-4.12.2/src/stats.c:550
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat at /workspace/srcdir/gap-4.12.2/src/stats.c:160
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:489
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:150 [inlined]
ExecProccall2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:224
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat at /workspace/srcdir/gap-4.12.2/src/stats.c:160
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:489
CALL_2ARGS at /workspace/srcdir/gap-4.12.2/src/calls.h:316 [inlined]
EvalOrExecCall at /workspace/srcdir/gap-4.12.2/src/funcs.c:150 [inlined]
EvalFunccall2args at /workspace/srcdir/gap-4.12.2/src/funcs.c:308
EVAL_EXPR at /workspace/srcdir/gap-4.12.2/src/exprs.h:98 [inlined]
ExecAssLVar at /workspace/srcdir/gap-4.12.2/src/vars.c:167
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat at /workspace/srcdir/gap-4.12.2/src/stats.c:160
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecForRangeHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:538 [inlined]
ExecForRange at /workspace/srcdir/gap-4.12.2/src/stats.c:550
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat at /workspace/srcdir/gap-4.12.2/src/stats.c:160
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
ExecSeqStatHelper at /workspace/srcdir/gap-4.12.2/src/stats.c:147 [inlined]
ExecSeqStat7 at /workspace/srcdir/gap-4.12.2/src/stats.c:190
EXEC_STAT at /workspace/srcdir/gap-4.12.2/src/stats.c:55 [inlined]
EXEC_CURR_FUNC at /workspace/srcdir/gap-4.12.2/src/stats.c:61
DoExecFunc at /workspace/srcdir/gap-4.12.2/src/funcs.c:468 [inlined]
DoExecFunc1args at /workspace/srcdir/gap-4.12.2/src/funcs.c:489
_call_gap_func at /home/runner/.julia/packages/GAP/ydeRk/src/ccalls.jl:322
call_gap_func_nokw at /home/runner/.julia/packages/GAP/ydeRk/src/ccalls.jl:288 [inlined]
GapObj at /home/runner/.julia/packages/GAP/ydeRk/src/ccalls.jl:299
unknown function (ip: 0x7fa0a2f918a5)
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
jl_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/julia.h:1966 [inlined]
do_call at /cache/build/default-amdci4-1/julialang/julia-master/src/interpreter.c:125
eval_value at /cache/build/default-amdci4-1/julialang/julia-master/src/interpreter.c:222
eval_stmt_value at /cache/build/default-amdci4-1/julialang/julia-master/src/interpreter.c:173 [inlined]
eval_body at /cache/build/default-amdci4-1/julialang/julia-master/src/interpreter.c:616
eval_body at /cache/build/default-amdci4-1/julialang/julia-master/src/interpreter.c:543
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-1/julialang/julia-master/src/interpreter.c:774
jl_toplevel_eval_flex at /cache/build/default-amdci4-1/julialang/julia-master/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/default-amdci4-1/julialang/julia-master/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/default-amdci4-1/julialang/julia-master/src/toplevel.c:993
eval at ./boot.jl:383 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
_include at ./loading.jl:2130
include at ./Base.jl:490 [inlined]
macro expansion at ./timing.jl:504 [inlined]
include at /home/runner/work/Oscar.jl/Oscar.jl/test/runtests.jl:76
#1040 at ./asyncmap.jl:94
unknown function (ip: 0x7fa0a2fbe4e0)
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
jl_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/julia.h:1966 [inlined]
do_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/builtins.c:768
#1056 at ./asyncmap.jl:228
unknown function (ip: 0x7fa0a2fbb612)
_jl_invoke at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:2890 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-1/julialang/julia-master/src/gf.c:3072
jl_apply at /cache/build/default-amdci4-1/julialang/julia-master/src/julia.h:1966 [inlined]
start_task at /cache/build/default-amdci4-1/julialang/julia-master/src/task.c:1238
Allocations: 281001018 (Pool: 2807[616](https://github.com/oscar-system/Oscar.jl/actions/runs/5516967428/jobs/10058976212?pr=2542#step:6:619)62; Big: 239356); GC: 142
ERROR: LoadError: Package Oscar errored during testing (received signal: 11)

The log is from this run:
https://github.com/oscar-system/Oscar.jl/actions/runs/5516967428/jobs/10058976212?pr=2542#step:6:441 (julia commit 680e3b3320f)

More logs:
https://github.com/oscar-system/Oscar.jl/actions/runs/5513751902/jobs/10052247798#step:6:811 (julia commit e2e34f6987d)
https://github.com/oscar-system/Oscar.jl/actions/runs/5511670816/jobs/10047525726#step:6:813 (julia 1.10.0-alpha1)

cc: @fingolfin

@benlorenz benlorenz added bug Something isn't working package: GAP nightly error with julia nightly or pre-releases labels Jul 11, 2023
@fingolfin
Copy link
Member

sigh

OK in this PR, we are in Julia code invoked from GAP, which worried me; but the other two are in pure GAP, it seems. (Not that this is great, but at least it's not yet another completely unrelated can of worms to be opened)

So I wonder if there is yet another GC change on the Julia side behind this...

@fingolfin
Copy link
Member

I am able to reproduce this with a loop:

for i in 1:100
  println(i)
  include("test/GAP/runtests.jl")
  println("-----")
end

It crashed in iteration 13 for me in Julia 1.10.0-alpha1 on an M1 Mac. So I'll see if I can bisect using this...

@fingolfin
Copy link
Member

Unfortunately it seems that the crash does not reproduce reliably (in subsequent tests with versions that exhibited the crash, so my hopes of being able to bisect don't seem to pan out :-(

@benlorenz
Copy link
Member Author

There is a slightly longer error message on a recent failure on macos here https://github.com/oscar-system/Oscar.jl/actions/runs/5541671673/jobs/10115375924#step:6:489. Maybe the ptr queue that is printed there can help?

To make sure it doesn't disappear I copied it here.

@fingolfin
Copy link
Member

I discovered JuliaLang/julia#50434 last night which was a gcext error that was fixed a few days ago in JuliaLang/julia#50533

Of course now we have even more nightly crashes for other reasons (???) so I am not sure whether or not this error here persists...

@benlorenz
Copy link
Member Author

benlorenz commented Jul 19, 2023

In the 32 CI jobs (here + here, all on Linux to avoid occupying the macos runners) that I ran last night on nightly there were 6 failures but none like the one discussed here.

Another round of tests is still running.

But there is one new issue: There are very long GC pauses now and some of the tests take extremely long.
For example the PolyhedralGeometry/timing test group takes 10-20 minutes instead of 30 seconds on 1.10. This testset needs to call GC.gc() twice for each of the 15 tests (plus a few more) and these explicit GC calls seem to cause 2 pauses, with a total of 10-30 seconds for each call.
Also the OscarInterface tests show some huge GC time:

  134992 ms (95287 ms GC) and 392MB allocated for alnuth/polynome.tst
-----------------------------------
total    338800 ms (253855 ms GC) and 416MB allocated
              0 failures in 4 files

I tried bisecting this locally which is always a bit complicated with timing issues as they don't always show up. But this seems to have started with JuliaLang/julia#50533 which added a new loop for the freelist to jl_gc_internal_obj_base_ptr which is called quite often by FindLiveRangeReverse from GAPs julia_gc.c. Looking at a few backtraces of a julia process that exhibits this behavior seems to confirm that these calls might be the reason.

(gdb) bt
#0  ijl_gc_internal_obj_base_ptr (p=0x7f4b)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:4004
#1  0x00007f4acaf3eb89 in FindLiveRangeReverse (end=0x7f4aa79fc000, start=<optimized out>, 
    arr=0x55f2aaf56290) at src/julia_gc.c:425
#2  SafeScanTaskStack (stack=stack@entry=0x55f2aaf56290, start=start@entry=0x7f4aa75fc000, 
    end=end@entry=0x7f4aa79fc000) at src/julia_gc.c:497
#3  0x00007f4acaf3fd87 in ScanTaskStack (end=<optimized out>, start=0x7f4aa75fc000, 
    task=0x7f4ac4450e20, rescan=1) at src/julia_gc.c:523
#4  GapTaskScanner (task=0x7f4ac4450e20, root_task=<optimized out>) at src/julia_gc.c:665
#5  0x00007f4b422a2440 in gc_mark_outrefs (meta_updated=<optimized out>, _new_obj=<optimized out>, 
    mq=<optimized out>, ptls=<optimized out>)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:2424
#6  gc_mark_loop_serial_ (ptls=ptls@entry=0x55f291c54260, mq=mq@entry=0x55f291c55dd8)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:2673
#7  0x00007f4b422a2c06 in gc_mark_loop_serial (ptls=ptls@entry=0x55f291c54260)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:2696
#8  0x00007f4b422a6709 in gc_mark_loop (ptls=0x55f291c54260)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:2809
#9  _jl_gc_collect (ptls=ptls@entry=0x55f291c54260, collection=collection@entry=JL_GC_AUTO)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:3128
#10 0x00007f4b422a8d4c in ijl_gc_collect (collection=JL_GC_FULL)
    at /home/lorenz/software/polymake/julia/julia/julia/src/gc.c:3452
#11 0x00007f4b41d426f8 in ?? ()

@benlorenz
Copy link
Member Author

benlorenz commented Jul 19, 2023

To trigger these long pauses it suffices to run some of our test files in the same session, e.g.:

$ julia-dev --project=.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.73 (2023-07-13)
 _/ |\__'_|_|_|\__'_|  |  Commit dcca46b756* (5 days old master)
|__/                   |

julia> GC.enable_logging(true); 

julia> GC.gc()
GC: pause 52.24ms. collected 31.875546MB. incr 

julia> GC.gc()
GC: pause 17.17ms. collected 0.032704MB. full recollect
GC: pause 45.19ms. collected 0.000000MB. incr 

julia> GC.gc()
GC: pause 10.83ms. collected 0.025824MB. full recollect
GC: pause 48.96ms. collected 0.002560MB. incr 

julia> using Oscar; using Test;
...<output removed>...

julia> Oscar.test_module("GAP/runtests"; new=false);
[ Info: Running tests for test/GAP/runtests.jl in same session
...<output removed>...

julia> GC.gc()
GC: pause 2805.88ms. collected 168.569744MB. full recollect
GC: pause 2754.58ms. collected 10.715928MB. incr 

julia> GC.gc()
GC: pause 5824.51ms. collected 22.815800MB. full recollect
GC: pause 2908.63ms. collected 0.176320MB. incr 

julia> GC.gc()
GC: pause 5845.09ms. collected 0.026720MB. full recollect
GC: pause 2940.97ms. collected 0.002864MB. incr 

Other testgroups should work as well, some more extreme, some less, e.g. for Combinatorics/runtests I just got these pauses afterwards:

julia> GC.gc()
GC: pause 3455.21ms. collected 0.026192MB. full recollect
GC: pause 2930.95ms. collected 0.002208MB. incr 

PS: This needs a current Oscar master, older releases are not working with julia nightly.

@fingolfin
Copy link
Member

Perhaps @d-netto or @vtjnash are interested in this?

@oscardssmith
Copy link

yeah if you can get a MWE of this, it would be great to get it into the GCBenchmarks

@fingolfin
Copy link
Member

The offending loop over the freelist was removed again in JuliaLang/julia#50599 so hopefully the long GC pauses are gone again...

@thofma
Copy link
Collaborator

thofma commented Jul 31, 2023

Should we close this in favor of #2441? Or maybe open a new issue? The CI situation with 1.10+? and nightly is absurd.

@benlorenz
Copy link
Member Author

I think the issue discussed here is resolved (the crash and the pauses).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working nightly error with julia nightly or pre-releases package: GAP
Projects
None yet
Development

No branches or pull requests

4 participants