1.11.0-alpha2: precompilation fails if `JULIA_DEPOT_PATH` is set #53983

matthias314 · 2024-04-06T22:33:48Z

This is a duplicate of #53831. I'm creating a new issue because I now have a reproducible MWE and because the old issue was apparently not noticed.

Setup: There is no ~/.julia, and JULIA_DEPOT_PATH is set to the empty directory /tmp/julia-depot. Then I try to install and use a package as follows:

~$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-alpha2 (2024-03-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.11.0-alpha2
Commit 9dfd28ab751 (2024-03-18 20:35 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i3-10110U CPU @ 2.10GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /tmp/julia-depot

(@v1.11) pkg> st
  Installing known registries into `/tmp/julia-depot`
Status `/tmp/julia-depot/environments/v1.11/Project.toml` (empty project)

(@v1.11) pkg> add Chairmarks
    Updating registry at `/tmp/julia-depot/registries/General.toml`
   Resolving package versions...
   Installed Chairmarks ─ v1.2.1
    Updating `/tmp/julia-depot/environments/v1.11/Project.toml`
  [0ca39b1e] + Chairmarks v1.2.1
    Updating `/tmp/julia-depot/environments/v1.11/Manifest.toml`
  [0ca39b1e] + Chairmarks v1.2.1
  [de0858da] + Printf v1.11.0
  [4ec0a83e] + Unicode v1.11.0
Precompiling project...
  ✓ Unicode
  ✓ Printf
  3 dependencies successfully precompiled in 3 seconds
  2 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions

julia> using Chairmarks
[ Info: Precompiling Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de] (cache misses: wrong dep version loaded (2))
┌ Warning: Module Printf with build ID fafbfcfd-be51-2905-0028-424d602318e3 is missing from the cache.
│ This may mean Printf [de0858da-6303-5e67-8744-51eddeeeb8d7] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2319
┌ Info: Skipping precompilation due to precompilable error. Importing Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de].
└   exception = nothing

julia>

The check marks (✓) in front of Unicode and Printf are actually red and not green.

Restarting:

~$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-alpha2 (2024-03-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> st
Status `/tmp/julia-depot/environments/v1.11/Project.toml`
  [0ca39b1e] Chairmarks v1.2.1

julia> using Chairmarks
[ Info: Precompiling Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de] (cache misses: wrong dep version loaded (2))
┌ Warning: Module Printf with build ID fafbfcfd-be51-2905-0028-424d602318e3 is missing from the cache.
│ This may mean Printf [de0858da-6303-5e67-8744-51eddeeeb8d7] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2319
┌ Info: Skipping precompilation due to precompilable error. Importing Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de].
└   exception = nothing

julia>

If JULIA_DEPOT_PATH is not set, then the problem disappears.

The text was updated successfully, but these errors were encountered:

vtjnash · 2024-04-07T01:16:57Z

It is a problem with #53326, which I was aware of there but didn't implement the handling for this

LilithHafner · 2024-06-06T14:40:04Z

@vtjnash, is this something you can take on?

vtjnash · 2024-06-06T16:01:36Z

Probably not in the short term right now

Fixes #53983 Unfortunately, TOML has a design problem with the way it is used by Pkg, so this does not yet work correctly. Need to finish changing uses of loaded_packages to explicit_loaded_modules in loading to try to make it less terribly broken.

Fixes #53983 Unfortunately, TOML has a design problem with the way it is used by Pkg, so this does not yet work correctly. Need to finish changing uses of loaded_packages to explicit_loaded_modules in loading to try to make it less terribly broken. Also currently one test is broken, as we need some more code rearrangement to make it legal again for an `__init__` method to import code.

Fixes #53983 This breaks one test, which relied on it being undefined behavior to rely on the execution of `__init__` having occurred before `using` is allowed to return. That usability issue is now (partially) eliminated.

This was an accidental late change in the PR, and I forgot why this needed to be exactly written this way (and why this wasn't equivalent). Followup to #54739 fixing #53983

robsmith11 · 2024-06-29T14:10:20Z

This seems to be broken still in 1.11.0-rc1 (which was released after this issue was resolved):

$ JULIA_DEPOT_PATH=/home/user/data/dot_julia /home/user/data/julia-1.11.0-rc1/bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-rc1 (2024-06-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] (cache misses: wrong dep version loaded (2))
┌ Warning: Module REPL with build ID fafbfcfd-9cbb-b77f-0014-e2674d4b007c is missing from the cache.
│ This may mean REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0].
└   exception = nothing
[ Info: Precompiling SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c] (cache misses: wrong dep version loaded (2))
┌ Warning: Module InteractiveUtils with build ID fafbfcfd-4d14-9bf7-0014-e2656d1ecfe3 is missing from the cache.
│ This may mean InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c].
└   exception = nothing
[ Info: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] (cache misses: wrong dep version loaded (2))
┌ Warning: Module InteractiveUtils with build ID fafbfcfd-4d14-9bf7-0014-e2656d1ecfe3 is missing from the cache.
│ This may mean InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8].
└   exception = nothing
[ Info: Precompiling PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d] (cache misses: wrong dep version loaded (2))
┌ Warning: Module Markdown with build ID fafbfcfd-2939-cd23-0014-e264ef460a86 is missing from the cache.
│ This may mean Markdown [d6f4376e-aef5-505a-96c1-9c027394607a] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d].
└   exception = nothing

matthias314 · 2024-06-29T14:26:18Z

Just for reference: the issue mentioned in #54739 (comment) also hasn't been fully resolved in 11.0-rc1. With JULIA_DEPOT_PATH set and a startup.jl containing the line using InteractiveUtils I get:

~$ /tmp/julia-1.11.0-rc1/bin/julia
┌ Warning: Replacing module `Base64`
└ @ Base loading.jl:2263
┌ Warning: Replacing module `Markdown`
└ @ Base loading.jl:2263
┌ Warning: Replacing module `InteractiveUtils`
└ @ Base loading.jl:2263
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-rc1 (2024-06-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>

More followup to fix issues with require. There was an accidental variable reuse (build_id) that caused it to be unable to load cache files in many cases. There was also missing check for a dependency already being loaded, resulting in trying to load it twice. Finally, the start_loading code may drop the require_lock, but the surrounding code was not prepared for that. Now integrate the necessary checks into start_loading, instead of needing to duplicate them before and afterwards. Fixes #53983 Fixes #54940 Closes #55064

This now allows the user to load any number of copies of a module, and uses the combination of the environment, explicitly loaded modules, and the requirements of the precompile caches to determine the meaning of a name and which files need to be loaded. Note however that package extensions continue to primarily only apply to the explicitly loaded modules, although they may get loaded incidentally as the dependency of another package, they won't get defined for every pair of combinations of triggering modules. Fixes #53983 (cherry picked from commit a1a2ac6)

More followup to fix issues with require. There was an accidental variable reuse (build_id) that caused it to be unable to load cache files in many cases. There was also missing check for a dependency already being loaded, resulting in trying to load it twice. Finally, the start_loading code may drop the require_lock, but the surrounding code was not prepared for that. Now integrate the necessary checks into start_loading, instead of needing to duplicate them before and afterwards. Fixes #53983 Fixes #54940 Closes #55064 (cherry picked from commit fba928d)

matthias314 mentioned this issue Apr 6, 2024

1.11.0-alpha2: problems with precompilation #53831

Closed

giordano added the packages Package management and loading label Apr 6, 2024

giordano added this to the 1.11 milestone Apr 6, 2024

vtjnash mentioned this issue Jun 8, 2024

finish implementation of upgradable stdlibs #54739

Merged

vtjnash closed this as completed in #54739 Jun 21, 2024

vtjnash closed this as completed in a1a2ac6 Jun 21, 2024

robsmith11 mentioned this issue Jul 5, 2024

1.11.0-rc1: precompilation fails if JULIA_DEPOT_PATH is set #55041

Closed

ViralBShah reopened this Jul 7, 2024

vtjnash mentioned this issue Jul 8, 2024

fix loading of repeated/concurrent modules #55066

Merged

vtjnash closed this as completed in #55066 Jul 11, 2024

vtjnash closed this as completed in fba928d Jul 11, 2024

matthias314 mentioned this issue Aug 2, 2024

precompilation problems with 1.11.0-rc2 #55340

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1.11.0-alpha2: precompilation fails if `JULIA_DEPOT_PATH` is set #53983

1.11.0-alpha2: precompilation fails if `JULIA_DEPOT_PATH` is set #53983

matthias314 commented Apr 6, 2024 •

edited

Loading

vtjnash commented Apr 7, 2024

LilithHafner commented Jun 6, 2024

vtjnash commented Jun 6, 2024

robsmith11 commented Jun 29, 2024

matthias314 commented Jun 29, 2024

1.11.0-alpha2: precompilation fails if JULIA_DEPOT_PATH is set #53983

1.11.0-alpha2: precompilation fails if JULIA_DEPOT_PATH is set #53983

Comments

matthias314 commented Apr 6, 2024 • edited Loading

vtjnash commented Apr 7, 2024

LilithHafner commented Jun 6, 2024

vtjnash commented Jun 6, 2024

robsmith11 commented Jun 29, 2024

matthias314 commented Jun 29, 2024

1.11.0-alpha2: precompilation fails if `JULIA_DEPOT_PATH` is set #53983

1.11.0-alpha2: precompilation fails if `JULIA_DEPOT_PATH` is set #53983

matthias314 commented Apr 6, 2024 •

edited

Loading