Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Package precompilation suddenly hangs on 1.5.4 without direct code changes #40457

Closed
evetion opened this issue Apr 13, 2021 · 3 comments
Closed

Package precompilation suddenly hangs on 1.5.4 without direct code changes #40457

evetion opened this issue Apr 13, 2021 · 3 comments

Comments

@evetion
Copy link
Sponsor Member

evetion commented Apr 13, 2021

As of a week ago, without code changes, package(s) hang on precompiling on 1.5.4. Julia 1.6.0 works fine.

The package in question is SpaceLiDAR.jl. The behavior can be replicated on the downstream package of GeoArrays.jl. Of interest is the CI build here: https://github.com/evetion/GeoArrays.jl/actions/runs/706154210 which was green two weeks ago, but when I now restart it, it will hang and timeout eventually (no code changes). The CPU usage is 100% for the Julia process.

If you interrupt the process with CTRL-C you get:

^CERROR: InterruptException:
Stacktrace:
 [1] poptask(::Base.InvasiveLinkedListSynchronized{Task}) at ./task.jl:704
 [2] wait at ./task.jl:712 [inlined]
 [3] wait(::Base.GenericCondition{Base.Threads.SpinLock}) at ./condition.jl:106
 [4] wait(::Base.Process) at ./process.jl:621
 [5] success at ./process.jl:483 [inlined]
 [6] compilecache(::Base.PkgId, ::String) at ./loading.jl:1287
 [7] _require(::Base.PkgId) at ./loading.jl:1030
 [8] require(::Base.PkgId) at ./loading.jl:928
 [9] require(::Module, ::Symbol) at ./loading.jl:923

After the interrupt and exiting Julia there's still a Julia process active taking 100%(!):

julia -Cnative -J/Applications/Julia-1.5.app/Contents/Resources/julia/lib/julia/sys.dylib -g1 -O0 --output-ji /Users/evetion/.julia/compiled/v1.5/SpaceLiDAR/jl_WJZ2yJ --output-incremental=yes --startup-file=no --history-file=no --warn-overwrite=yes --color=yes

If I kill that process with a SIGTERM, I get:

signal (15): Terminated: 15
in expression starting at /Users/evetion/code/SpaceLiDAR/src/SpaceLiDAR.jl:30   # this is `precompile(GeoArrays.read, (String,))`
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
__psynch_cvwait at /usr/lib/system/libsystem_kernel.dylib (unknown line)
unknown function (ip: 0x0)
unalias_unionall at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:801
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1020
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:739
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1162 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1317
subtype_tuple_varargs at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:944 [inlined]
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1043
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:708
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:739
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1162 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1317
subtype_tuple_varargs at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:943 [inlined]
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1043
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:708
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:739
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1162 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1317
subtype_tuple_varargs at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:944 [inlined]
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1043
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:708
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
with_tvar at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:739
subtype_tuple_tail at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:0
subtype_tuple at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1162 [inlined]
subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1317
exists_subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1420 [inlined]
forall_exists_subtype at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1448
jl_subtype_env at /Users/julia/buildbot/worker/package_macos64/build/src/subtype.c:1877
jl_f_issubtype at /Users/julia/buildbot/worker/package_macos64/build/src/builtins.c:446
⊑ at ./compiler/typelattice.jl:177
tmerge at ./compiler/typelimits.jl:288
typeinf_local at ./compiler/abstractinterpretation.jl:1220
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_apply at ./compiler/abstractinterpretation.jl:635
abstract_call_known at ./compiler/abstractinterpretation.jl:704
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_edge at ./compiler/typeinfer.jl:484
abstract_call_method at ./compiler/abstractinterpretation.jl:419
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:111
abstract_call_known at ./compiler/abstractinterpretation.jl:904
abstract_call at ./compiler/abstractinterpretation.jl:926
abstract_call at ./compiler/abstractinterpretation.jl:911
abstract_eval at ./compiler/abstractinterpretation.jl:1005
typeinf_local at ./compiler/abstractinterpretation.jl:1270
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1326
typeinf at ./compiler/typeinfer.jl:12
typeinf_ext at ./compiler/typeinfer.jl:570
typeinf_ext at ./compiler/typeinfer.jl:601
jfptr_typeinf_ext_22058.clone_1 at /Applications/Julia-1.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:1691 [inlined]
jl_type_infer at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:296
_generate_from_hint at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2076 [inlined]
jl_compile_now at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2094
jl_compile_hint at /Users/julia/buildbot/worker/package_macos64/build/src/gf.c:2112
precompile at ./essentials.jl:460
unknown function (ip: 0x146242cbb)
jl_apply at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:1691 [inlined]
do_call at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:117
eval_body at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:0
jl_interpret_toplevel_thunk at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:840
jl_eval_module_expr at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:197
jl_toplevel_eval_flex at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:666
jl_parse_eval_all at /Users/julia/buildbot/worker/package_macos64/build/src/ast.c:913
jl_load_rewrite at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:914 [inlined]
jl_load at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:919
include at ./Base.jl:380
include at ./Base.jl:368
jl_apply at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:1691 [inlined]
do_call at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:117
eval_body at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:0
eval_body at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:492
jl_interpret_toplevel_thunk at /Users/julia/buildbot/worker/package_macos64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:840
jl_toplevel_eval at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:849 [inlined]
jl_toplevel_eval_in at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:883
eval at ./boot.jl:347 [inlined]
eval at ./client.jl:467
top-level scope at ./none:3
jl_toplevel_eval_flex at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:834
jl_toplevel_eval_flex at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:790
jl_toplevel_eval at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:849 [inlined]
jl_toplevel_eval_in at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:883
eval at ./boot.jl:347
exec_options at ./client.jl:272
_start at ./client.jl:506
jfptr__start_69719.clone_1 at /Applications/Julia-1.5.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
true_main at /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia (unknown line)
main at /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia (unknown line)
unknown function (ip: 0x0)
Allocations: 670788573 (Pool: 670750822; Big: 37751); GC: 593

I've tried to compare package dependency differences, but didn't find any direct dependencies that changed that would explain the behavior. Asking around in Slack, we found similar behaviour (CI timeout while precompiling) at CliMA/Oceananigans.jl#1561, that started happening around the same time. The only overlap is ChainRulesCore 0.9.37, but when I pin a lower version for GeoArrays (https://github.com/evetion/GeoArrays.jl/pull/69/checks?check_run_id=2330978888), it still hangs.

There's a lot going on, not sure what is relevant and what not. I'd like to replicate and pinpoint this a bit better, but I'm at a loss on how to tackle this.
@glwagner glwagner mentioned this issue Apr 21, 2021
Closed
@ali-ramadhan ali-ramadhan mentioned this issue Apr 22, 2021
Open
4 tasks
@glwagner glwagner mentioned this issue Apr 27, 2021
Closed
@glwagner
Copy link

This could be due to changes in ArrayInterface; see JuliaArrays/ArrayInterface.jl#142

glwagner referenced this issue in CliMA/Oceananigans.jl Apr 27, 2021
@glwagner
Pin ArrayInterface at 3.1.6 (#1621)
755e9a7 
* Pins ArrayInterface at 3.1.6

* Bump to 0.56.1

* Update Project.toml

* Update compat for ArrayInterface to "3.1.6"

* Actually specify strict compat for ArrayInterface at 3.1.6
@JuliaRegistrator JuliaRegistrator mentioned this issue Apr 27, 2021
Merged
@glwagner glwagner mentioned this issue Apr 27, 2021
Merged
@evetion
Copy link
Sponsor Member Author

evetion commented Apr 27, 2021

I can confirm pinning ArrayInterface to 3.1.6 fixes this locally.

@evetion
Copy link
Sponsor Member Author

evetion commented Apr 29, 2021

The newly released ArrayInterface 3.1.9 also fixes this, will close this issue now.

To summarize, a small interface package, ArrayInterface, used indirectly by many different packages quietly introduced a bug (only on Julia 1.5) that caused a hang of Julia. For the package authors involved it was not possible to directly determine what (package) caused the hang.

@evetion evetion closed this as completed Apr 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@evetion @glwagner