Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Precompiling too many pkgs in parallel crash or throw on memory-limited Windows #53141

Open
inkydragon opened this issue Feb 1, 2024 · 2 comments
Open

Precompiling too many pkgs in parallel crash or throw on memory-limited Windows #53141

inkydragon opened this issue Feb 1, 2024 · 2 comments

Comments

@inkydragon
Copy link
Sponsor Member

Context: When trying to reproduce #53139, Julia v1.10 crash or throw errors in Windows Sandbox.

This may not be a bug or even a Windows-Only issue.
What might be worth improving here is to determine the number of Precompiling parallels based on both memory size and number of CPU cores.

env

  • Host: Win 11 Pro 23H2, 22631.3007

  • A new empty Win Sandbox, Win 11 Enterprise 22H2, 22621.3007

    • CPU core x32
    • Memory: 4GB, Available in idle state is 2.7GB

  • Julia v1.10.0 x64, download from https://julialang.org/downloads/
    Installed with Admin rights.

    versioninfo 
     julia> versioninfo()
 Julia Version 1.10.0
 Commit 3120989f39 (2023-12-25 18:01 UTC)
 Build Info:
   Official https://julialang.org/ release
 Platform Info:
   OS: Windows (x86_64-w64-mingw32)
   CPU: 32 × 13th Gen Intel(R) Core(TM) i9-13900HX
   WORD_SIZE: 64
   LIBM: libopenlibm
   LLVM: libLLVM-15.0.7 (ORCJIT, goldmont)
   Threads: 1 on 32 virtual cores

Note: Since the host shares the same CPU configuration as the sandbox, but the sandbox is limited in memory size, it is possible to compile too many packages in parallel when pre-compiling packages.

Steps to Reproduce

  1. open julia

  2. ]add CairoMakie

  3. failded in precompile
    image

  4. Continue ]precompile CairoMakie

  5. Silent crash or EXCEPTION_ACCESS_VIOLATION or ReadOnlyMemoryError() or OutOfMemoryError() or ...

EXCEPTION_ACCESS_VIOLATION 
┌ AbstractLattices
│  Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
│  Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffae9010dc1 -- set_page_metadata at C:/workdir/src\gc.h:330 [inlined]
│  gc_add_page at C:/workdir/src\gc.c:1274
│  in expression starting at none:0
│  set_page_metadata at C:/workdir/src\gc.h:330 [inlined]
│  gc_add_page at C:/workdir/src\gc.c:1274
│  jl_gc_pool_alloc_inner at C:/workdir/src\gc.c:1329
│  jl_gc_pool_alloc_noinline at C:/workdir/src\gc.c:1350 [inlined]
│  jl_gc_alloc_ at C:/workdir/src\julia_internal.h:477 [inlined]
│  jl_gc_alloc at C:/workdir/src\gc.c:3570
│  _new_array_ at C:/workdir/src\array.c:134
│  _new_array at C:/workdir/src\array.c:198 [inlined]
│  ijl_alloc_array_1d at C:/workdir/src\array.c:436 [inlined]
│  ijl_alloc_vec_any at C:/workdir/src\array.c:534
└
┌ IteratorInterfaceExtensions
│  Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
│  Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffae9010dc1 -- set_page_metadata at C:/workdir/src\gc.h:330 [inlined]
│  gc_add_page at C:/workdir/src\gc.c:1274
│  in expression starting at none:0
│  set_page_metadata at C:/workdir/src\gc.h:330 [inlined]
│  gc_add_page at C:/workdir/src\gc.c:1274
│  jl_gc_pool_alloc_inner at C:/workdir/src\gc.c:1329
│  jl_gc_pool_alloc_noinline at C:/workdir/src\gc.c:1350 [inlined]
│  jl_gc_alloc_ at C:/workdir/src\julia_internal.h:477 [inlined]
│  jl_gc_alloc at C:/workdir/src\gc.c:3570
│  _new_array_ at C:/workdir/src\array.c:134
│  _new_array at C:/workdir/src\array.c:198 [inlined]
│  ijl_alloc_array_1d at C:/workdir/src\array.c:436 [inlined]
│  ijl_alloc_vec_any at C:/workdir/src\array.c:534
│  ijl_idtable_rehash at C:/workdir/src\iddict.c:18
│  jl_restore_system_image_from_stream_ at C:/workdir/src\staticdata.c:3263
│  jl_restore_system_image_from_stream at C:/workdir/src\staticdata.c:3458 [inlined]
│  ijl_restore_system_image_data at C:/workdir/src\staticdata.c:3523
│  jl_load_sysimg_so at C:/workdir/src\staticdata.c:586 [inlined]
│  ijl_restore_system_image at C:/workdir/src\staticdata.c:3494
│  _finish_julia_init at C:/workdir/src\init.c:859
│  jl_repl_entrypoint at C:/workdir/src\jlapi.c:725
│  mainCRTStartup at C:/workdir/cli\loader_exe.c:58
│  BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line)
│  RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
│  Allocations: 0 (Pool: 0; Big: 0); GC: 0
└
┌ AbstractFFTs
│  Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
│  Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffae90769f2 -- unknown function (ip: 00007ffae90769f2)
│  in expression starting at none:0
│  unknown function (ip: 00007ffae90769f2)
│  unknown function (ip: 00007ffae90777ac)
│  unknown function (ip: 00007ffae8fc0f68)
│  unknown function (ip: 00007ffae8fc2e2d)
│  unknown function (ip: 00007ffae8fda760)
│  unknown function (ip: 00007ffae902804c)
└
ReadOnlyMemoryError 
┌ JLLWrappers
│  Internal error: encountered unexpected error in runtime:
│  ReadOnlyMemoryError()
│  jl_print_task_backtraces at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_gc_collect at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_gc_alloc at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_gdb_dump_threadinfo at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_alloc_array_1d at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad3f13c4c)
│  unknown function (ip: 00007ffad4d3ae89)
│  unknown function (ip: 00007ffad4d3ac54)
│  unknown function (ip: 00007ffad3f13c82)
│  unknown function (ip: 00007ffad4a1d2fc)
│  unknown function (ip: 00007ffad457137b)
│  unknown function (ip: 00007ffad367f106)
│  unknown function (ip: 00007ffad50a00d4)
│  unknown function (ip: 00007ffad37b6740)
│  unknown function (ip: 00007ffad3f121d4)
│  unknown function (ip: 00007ffad51bd6b5)
│  unknown function (ip: 00007ffad43aa4a3)
│  unknown function (ip: 00007ffad4e3622a)
│  unknown function (ip: 00007ffad459f3a7)
│  unknown function (ip: 00007ffad4207633)
│  unknown function (ip: 00007ffad4d3bba3)
│  unknown function (ip: 00007ffad3a5deb8)
│  unknown function (ip: 00007ffad3a61a9d)
│  unknown function (ip: 00007ffad4a17675)
│  unknown function (ip: 00007ffad4cfc86c)
│  unknown function (ip: 00007ffad3e30ac8)
│  unknown function (ip: 00007ffad367ea83)
│  unknown function (ip: 00007ffad50a00d4)
│  unknown function (ip: 00007ffad37b6740)
│  unknown function (ip: 00007ffad3f121d4)
│  unknown function (ip: 00007ffad51bd6b5)
│  unknown function (ip: 00007ffad43aa4a3)
│  unknown function (ip: 00007ffad4e3622a)
│  unknown function (ip: 00007ffad459f3a7)
│  unknown function (ip: 00007ffad4207633)
│  unknown function (ip: 00007ffad4d3bba3)
│  unknown function (ip: 00007ffad3a5deb8)
│  unknown function (ip: 00007ffad3a61a9d)
│  unknown function (ip: 00007ffad4a17675)
│  unknown function (ip: 00007ffad4cfc86c)
│  unknown function (ip: 00007ffad3e30ac8)
│  unknown function (ip: 00007ffad367ea83)
│  unknown function (ip: 00007ffad50a00d4)
│  unknown function (ip: 00007ffad37b6740)
│  unknown function (ip: 00007ffad3f121d4)
│  unknown function (ip: 00007ffad51bd6b5)
│  unknown function (ip: 00007ffad43aa4a3)
│  unknown function (ip: 00007ffad4e3622a)
│  unknown function (ip: 00007ffad459f3a7)
│  unknown function (ip: 00007ffad4207633)
│  unknown function (ip: 00007ffad4d3bba3)
│  unknown function (ip: 00007ffad3a5deb8)
│  unknown function (ip: 00007ffad3a61a9d)
│  unknown function (ip: 00007ffad4a17675)
│  unknown function (ip: 00007ffad4cfc86c)
│  unknown function (ip: 00007ffad3e30ac8)
│  unknown function (ip: 00007ffad367ea83)
│  unknown function (ip: 00007ffad50a00d4)
│  unknown function (ip: 00007ffad3b76d6d)
│  unknown function (ip: 00007ffad4581ad6)
│  unknown function (ip: 00007ffad4687726)
│  unknown function (ip: 00007ffad4e17a3a)
│  jl_type_infer at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_generate_fptr_impl at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-codegen.dll (unknown line)
│  jl_normalize_to_compilable_mi at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_apply_generic at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad3ae6819)
│  unknown function (ip: 00007ffad496e11e)
│  jl_f_set_binding_type at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad4e285a7)
│  unknown function (ip: 00007ffad45cfeb2)
│  unknown function (ip: 00007ffad510b988)
│  unknown function (ip: 00007ffad3f29982)
│  unknown function (ip: 00007ffad4a7c0a2)
│  unknown function (ip: 00007ffad3bbb2e0)
│  jl_f__call_in_world at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_f_set_binding_type at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad3ae6c43)
│  jl_f_set_binding_type at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad36ee95a)
│  jl_f_set_binding_type at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad4d7362e)
│  jl_f_set_binding_type at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad4980d82)
│  unknown function (ip: 00007ffad495d70b)
│  unknown function (ip: 00007ffad45b60cd)
│  unknown function (ip: 00007ffad4a2c8da)
│  unknown function (ip: 00007ffad3a91511)
│  unknown function (ip: 00007ffad3a91544)
│  jl_f__call_latest at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_fptr_interpret_call at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  include at C:\Users\WDAGUtilityAccount\.julia\packages\JLLWrappers\pG9bm\src\JLLWrappers.jl:1
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_interpret_toplevel_thunk at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  top-level scope at C:\Users\WDAGUtilityAccount\.julia\packages\JLLWrappers\pG9bm\src\wrapper_generators.jl:1
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_toplevel_eval_in at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad45b6241)
│  unknown function (ip: 00007ffad4a2c8da)
│  unknown function (ip: 00007ffad3a91511)
│  unknown function (ip: 00007ffad3a91544)
│  jl_f__call_latest at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_fptr_interpret_call at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  include at C:\Users\WDAGUtilityAccount\.julia\packages\JLLWrappers\pG9bm\src\JLLWrappers.jl:1
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_interpret_toplevel_thunk at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  top-level scope at C:\Users\WDAGUtilityAccount\.julia\packages\JLLWrappers\pG9bm\src\JLLWrappers.jl:28
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_toplevel_eval_in at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad45b6241)
│  unknown function (ip: 00007ffad4a2c8da)
│  unknown function (ip: 00007ffad4e88eb7)
│  unknown function (ip: 00007ffad46d1c21)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_init_restored_module at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_interpret_toplevel_thunk at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  top-level scope at stdin:3
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_toplevel_eval_flex at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  ijl_toplevel_eval_in at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  unknown function (ip: 00007ffad45b6241)
│  unknown function (ip: 00007ffad52c5db6)
│  unknown function (ip: 00007ffad4f3d6b3)
│  unknown function (ip: 00007ffad43f9f7c)
│  ijl_call2 at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  jl_repl_entrypoint at C:\Users\WDAGUtilityAccount\AppData\Local\Programs\Julia-1.10.0\bin\libjulia-internal.dll (unknown line)
│  mainCRTStartup at C:/workdir/cli\loader_exe.c:58
│  BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line)
│  RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
└
The paging file is too small 
┌ StatsAPI
│  fatal: error thrown and no exception handler available.
│  InitError(mod=:OpenBLAS_jll, error=ErrorException("could not load library "libopenblas64_.dll"
│  The paging file is too small for this operation to complete. "))
│  ijl_errorf at C:/workdir/src\rtutils.c:77
│  ijl_load_dynamic_library at C:/workdir/src\dlload.c:388
│  #dlopen#3 at .\libdl.jl:117
│  dlopen at .\libdl.jl:116 [inlined]
│  dlopen at .\libdl.jl:116 [inlined]
│  __init__ at C:\workdir\usr\share\julia\stdlib\v1.10\OpenBLAS_jll\src\OpenBLAS_jll.jl:70
└
OutOfMemoryError() 
┌ StatsAPI
│  Internal error: encountered unexpected error in runtime:
│  OutOfMemoryError()
│  gc_perm_alloc_large at C:/workdir/src\gc.c:3935
│  jl_gc_big_alloc_inner at C:/workdir/src\gc.c:3570 [inlined]
│  jl_gc_big_alloc_noinline at C:/workdir/src\gc.c:1045 [inlined]
│  jl_gc_alloc_ at C:/workdir/src\julia_internal.h:482 [inlined]
│  jl_gc_alloc at C:/workdir/src\gc.c:3570
│  jl_gc_alloc_buf at C:/workdir/src\julia_internal.h:515 [inlined]
│  jl_gc_alloc_buf at C:/workdir/src\julia_internal.h:513 [inlined]
│  array_resize_buffer at C:/workdir/src\array.c:698
│  jl_array_grow_at_end at C:/workdir/src\array.c:893
│  ijl_array_grow_end at C:/workdir/src\array.c:955 [inlined]
│  ijl_array_ptr_1d_push at C:/workdir/src\array.c:1259
│  jl_add_method_root at C:/workdir/src\method.c:1212
│  literal_val_id at C:/workdir/src\ircode.c:86 [inlined]
│  jl_encode_as_indexed_root at C:/workdir/src\ircode.c:106
│  jl_encode_value_ at C:/workdir/src\ircode.c:358
│  ijl_compress_ir at C:/workdir/src\ircode.c:814
└
@inkydragon
Copy link
Sponsor Member Author

inkydragon commented Feb 1, 2024 •
edited
Loading

Might reopen #39396

    # Windows sometimes hits a ReadOnlyMemoryError, so we halve the default number of tasks. Issue JuliaLang/Pkg.jl#2323
    # TODO: Investigate why this happens in windows and restore the full task limit
    default_num_tasks = Sys.iswindows() ? div(Sys.CPU_THREADS::Int, 2) + 1 : Sys.CPU_THREADS::Int + 1

https://github.com/JuliaLang/Pkg.jl/blob/f3b81f1aac77acf08f5d847ead29ad0a228dec67/src/API.jl#L1097-L1099

@nsajko
Copy link
Contributor

nsajko commented Feb 1, 2024

This happened to me a few times, on Linux, after which I set JULIA_NUM_PRECOMPILE_TASKS to three.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nsajko @inkydragon