Skip to content

Move compiled cache from DATAROOTDIR to LIBDIR #54353

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
+38 −28
Open

Move compiled cache from DATAROOTDIR to LIBDIR #54353

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3e85612
Move compiled cache from DATAROOTDIR to LIBDIR
nalimilan May 4, 2024
476e15d
Merge branch 'master' into nl/compiled
nalimilan Nov 2, 2025
1def6ec
Tentative fix
nalimilan Nov 6, 2025
1a3311c
WIP
nalimilan Nov 7, 2025
f98511b
Merge remote-tracking branch 'origin/master' into nl/compiled
nalimilan Nov 9, 2025
5cdd017
Fix relocatedepot test
nalimilan Nov 10, 2025
879f1df
Fix
nalimilan Nov 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,8 @@ ifeq ($(JULIA_BUILD_MODE),release)
else ifeq ($(JULIA_BUILD_MODE),debug)
$(INSTALL_M) $(build_private_libdir)/sys-debug.$(SHLIB_EXT) $(DESTDIR)$(private_libdir)
endif
# Copy depot
cp -R -L $(build_private_libdir)/compiled/ $(DESTDIR)$(private_libdir)

# Copy in all .jl sources as well
mkdir -p $(DESTDIR)$(datarootdir)/julia/base $(DESTDIR)$(datarootdir)/julia/test
Expand Down
28 changes: 20 additions & 8 deletions base/initdefs.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,20 @@ environments, repo clones, cached compiled package images, and configuration
files. By default it includes:

1. `~/.julia` where `~` is the user home as appropriate on the system;
2. an architecture-specific shared system directory, e.g. `/usr/local/share/julia`;
3. an architecture-independent shared system directory, e.g. `/usr/share/julia`.

So `DEPOT_PATH` might be:
2. an architecture-specific shared system directory specific to the local host,
e.g. `/usr/local/lib/julia`;
3. an architecture-independent shared system directory specific to the local host,
e.g. `/usr/local/share/julia`;
Comment on lines +53 to +56
Copy link
Member Author

@nalimilan nalimilan May 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure it's a good idea to look in /usr/local by default. I've kept /usr/local/share/ for backward compatibility, and added /usr/local/lib for consistency. But this doesn't make a lot of sense as the next two dirs (4 and 5) are relative to the julia path so they will correspond to /usr/local/ if Julia was installed there. And if there are two Julia copies, one in /usr and the other in /usr/local, the latter is going to interfere with the former.

4. an architecture-specific shared system directory,
e.g. `/usr/lib/julia`;
5. an architecture-independent shared system directory,
e.g. `/usr/share/julia`

All directories except the first are relative to the path of the `julia` executable.
So if Julia is installed to `/usr/bin/julia`, `DEPOT_PATH` might be:
```julia
[joinpath(homedir(), ".julia"), "/usr/local/share/julia", "/usr/share/julia"]
[joinpath(homedir(), ".julia"), "/usr/local/lib/julia", "/usr/local/share/julia",
"/usr/lib/julia", "/usr/share/julia"]
```
The first entry is the "user depot" and should be writable by and owned by the
current user. The user depot is where: registries are cloned, new package versions
Expand All @@ -76,7 +84,7 @@ Here is an overview of some of the subdirectories that may exist in a depot:
* `artifacts`: Contains content that packages use for which Pkg manages the installation of.
* `clones`: Contains full clones of package repos. Maintained by `Pkg.jl` and used as a cache.
* `config`: Contains julia-level configuration such as a `startup.jl`.
* `compiled`: Contains precompiled `*.ji` files for packages. Maintained by Julia.
* `compiled`: Contains precompiled `*.ji` and `*.so` files for packages. Maintained by Julia.
* `dev`: Default directory for `Pkg.develop`. Maintained by `Pkg.jl` and the user.
* `environments`: Default package environments. For instance the global environment for a specific julia version. Maintained by `Pkg.jl`.
* `logs`: Contains logs of `Pkg` and `REPL` operations. Maintained by `Pkg.jl` and Julia.
Expand All @@ -95,9 +103,13 @@ See also [`JULIA_DEPOT_PATH`](@ref JULIA_DEPOT_PATH), and
const DEPOT_PATH = String[]

function append_bundled_depot_path!(DEPOT_PATH)
path = abspath(Sys.BINDIR, "..", "local", "share", "julia")
path = abspath(Sys.BINDIR, "..", "local", "foo", LIBDIR, "julia")
path in DEPOT_PATH || push!(DEPOT_PATH, path)
path = abspath(Sys.BINDIR, "..", "local", "foo", DATAROOTDIR, "julia")
path in DEPOT_PATH || push!(DEPOT_PATH, path)
path = abspath(Sys.BINDIR, LIBDIR, "julia")
path in DEPOT_PATH || push!(DEPOT_PATH, path)
path = abspath(Sys.BINDIR, Base.DATAROOTDIR, "julia")
path = abspath(Sys.BINDIR, DATAROOTDIR, "julia")
path in DEPOT_PATH || push!(DEPOT_PATH, path)
return DEPOT_PATH
end
Expand Down
10 changes: 4 additions & 6 deletions pkgimage.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,8 @@ JULIAHOME := $(SRCDIR)
include $(JULIAHOME)/Make.inc
include $(JULIAHOME)/stdlib/stdlib.mk

DEPOTDIR := $(build_prefix)/share/julia

# set some influential environment variables
export JULIA_DEPOT_PATH := $(shell echo $(call cygpath_w,$(DEPOTDIR)))
export JULIA_DEPOT_PATH := $(shell echo $(call cygpath_w,$(build_private_libdir)))$(PATHSEP)$(shell echo $(call cygpath_w,$(build_datarootdir)/julia))
export JULIA_LOAD_PATH := @stdlib$(PATHSEP)$(shell echo $(call cygpath_w,$(JULIAHOME)/stdlib))
unexport JULIA_PROJECT :=
unexport JULIA_BINDIR :=
Expand All @@ -19,13 +17,13 @@ release: $(BUILDDIR)/stdlib/release.image
debug: $(BUILDDIR)/stdlib/debug.image
all: release debug

$(DEPOTDIR)/compiled:
$(build_private_libdir)/compiled:
mkdir -p $@

print-depot-path:
@$(call PRINT_JULIA, $(call spawn,$(JULIA_EXECUTABLE)) --startup-file=no -e '@show Base.DEPOT_PATH')

$(BUILDDIR)/stdlib/%.image: $(JULIAHOME)/stdlib/Project.toml $(JULIAHOME)/stdlib/Manifest.toml $(INDEPENDENT_STDLIBS_SRCS) $(DEPOTDIR)/compiled
$(BUILDDIR)/stdlib/%.image: $(JULIAHOME)/stdlib/Project.toml $(JULIAHOME)/stdlib/Manifest.toml $(INDEPENDENT_STDLIBS_SRCS) $(build_private_libdir)/compiled
@$(call PRINT_JULIA, JULIA_CPU_TARGET="sysimage" $(call spawn,$(JULIA_EXECUTABLE)) --startup-file=no -e \
'Base.Precompilation.precompilepkgs(configs=[``=>Base.CacheFlags(debug_level=2, opt_level=3), ``=>Base.CacheFlags(check_bounds=1, debug_level=2, opt_level=3)])')
touch $@
Expand All @@ -34,5 +32,5 @@ $(BUILDDIR)/stdlib/release.image: $(build_private_libdir)/sys.$(SHLIB_EXT)
$(BUILDDIR)/stdlib/debug.image: $(build_private_libdir)/sys-debug.$(SHLIB_EXT)

clean:
rm -rf $(DEPOTDIR)/compiled
rm -rf $(build_private_libdir)/compiled
rm -f $(BUILDDIR)/stdlib/*.image
8 changes: 5 additions & 3 deletions test/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ VERSDIR := v$(shell cut -d. -f1-2 < $(JULIAHOME)/VERSION)
STDLIBDIR := $(build_datarootdir)/julia/stdlib/$(VERSDIR)
# TODO: this Makefile ignores BUILDDIR, except for computing JULIA_EXECUTABLE

export JULIA_DEPOT_PATH := $(build_prefix)/share/julia
export JULIA_DEPOT_PATH := $(shell echo $(call cygpath_w,$(build_private_libdir)))
export JULIA_LOAD_PATH := @$(PATHSEP)@stdlib
unexport JULIA_PROJECT :=
unexport JULIA_BINDIR :=
Expand Down Expand Up @@ -46,7 +46,8 @@ relocatedepot:
@cd $(SRCDIR) && \
$(call PRINT_JULIA, $(call spawn,$(JULIA_EXECUTABLE)) $(TEST_JULIA_OPTIONS) ./runtests.jl $(TEST_SCRIPT_OPTIONS) $@)
@mkdir $(SRCDIR)/relocatedepot
@cp -R $(build_datarootdir)/julia $(SRCDIR)/relocatedepot
@cp -R $(build_datarootdir)/julia/* $(SRCDIR)/relocatedepot
@cp -R $(build_private_libdir)/compiled $(SRCDIR)/relocatedepot
@cp -R $(SRCDIR)/RelocationTestPkg1 $(SRCDIR)/relocatedepot
@cp -R $(SRCDIR)/RelocationTestPkg2 $(SRCDIR)/relocatedepot
@cp -R $(SRCDIR)/RelocationTestPkg3 $(SRCDIR)/relocatedepot
Expand All @@ -59,7 +60,8 @@ revise-relocatedepot: revise-% :
@cd $(SRCDIR) && \
$(call PRINT_JULIA, $(call spawn,$(JULIA_EXECUTABLE)) $(TEST_JULIA_OPTIONS) ./runtests.jl $(TEST_SCRIPT_OPTIONS) --revise $*)
@mkdir $(SRCDIR)/relocatedepot
@cp -R $(build_datarootdir)/julia $(SRCDIR)/relocatedepot
@cp -R $(build_datarootdir)/julia/* $(SRCDIR)/relocatedepot
@cp -R $(build_private_libdir)/compiled $(SRCDIR)/relocatedepot
@cp -R $(SRCDIR)/RelocationTestPkg1 $(SRCDIR)/relocatedepot
@cp -R $(SRCDIR)/RelocationTestPkg2 $(SRCDIR)/relocatedepot
@cp -R $(SRCDIR)/RelocationTestPkg3 $(SRCDIR)/relocatedepot
Expand Down
2 changes: 1 addition & 1 deletion test/embedding/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ EXE := $(suffix $(abspath $(JULIA)))

# get compiler and linker flags. (see: `contrib/julia-config.jl`)
JULIA_CONFIG := $(JULIA) -e 'include(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia", "julia-config.jl"))' --
JULIA_LIBDIR := $(shell $(JULIA) -e 'println(joinpath(Sys.BINDIR, "..", "lib"))' --)
JULIA_LIBDIR := $(shell $(JULIA) -e 'println(joinpath(Sys.BINDIR, Base.LIBDIR))' --)
CPPFLAGS_ADD :=
CFLAGS_ADD = $(shell $(JULIA_CONFIG) --cflags)
LDFLAGS_ADD = -lm $(shell $(JULIA_CONFIG) --ldflags --ldlibs)
Expand Down
14 changes: 5 additions & 9 deletions test/relocatedepot.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ else
@testset "load stdlib from test/relocatedepot" begin
test_harness() do
push!(LOAD_PATH, "@stdlib")
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot", "julia"))
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot"))
# stdlib should be already precompiled
pkg = Base.identify_package("DelimitedFiles")
@test Base.isprecompiled(pkg) == true
Expand All @@ -264,8 +264,7 @@ else
pkgname = "RelocationTestPkg1"
test_harness() do
push!(LOAD_PATH, joinpath(@__DIR__, "relocatedepot"))
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot")) # required to find src files
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot", "julia")) # contains cache file
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot"))
pkg = Base.identify_package(pkgname)
@test Base.isprecompiled(pkg) == true
@test Base.isrelocatable(pkg) == true
Expand All @@ -276,8 +275,7 @@ else
pkgname = "RelocationTestPkg2"
test_harness() do
push!(LOAD_PATH, joinpath(@__DIR__, "relocatedepot"))
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot")) # required to find src files
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot", "julia")) # contains cache file
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot"))
pkg = Base.identify_package(pkgname)
@test Base.isprecompiled(pkg) == false # moving depot changes mtime of include_dependency
@test Base.isrelocatable(pkg) == false # because not precompiled
Expand All @@ -294,8 +292,7 @@ else
pkgname = "RelocationTestPkg3"
test_harness() do
push!(LOAD_PATH, joinpath(@__DIR__, "relocatedepot"))
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot")) # required to find src files
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot", "julia")) # contains cache file
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot"))
pkg = Base.identify_package(pkgname)
@test Base.isprecompiled(pkg) == true
@test Base.isrelocatable(pkg) == true
Expand All @@ -309,8 +306,7 @@ else
pkgname = "RelocationTestPkg4"
test_harness() do
push!(LOAD_PATH, @__DIR__, "relocatedepot")
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot")) # required to find src files
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot", "julia")) # contains cache file
push!(DEPOT_PATH, joinpath(@__DIR__, "relocatedepot"))
pkg = Base.identify_package(pkgname)
# precompiled but not relocatable
@test Base.isprecompiled(pkg) == true
Expand Down
2 changes: 1 addition & 1 deletion test/trimming/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ EXE := $(suffix $(abspath $(JULIA)))

# get compiler and linker flags. (see: `contrib/julia-config.jl`)
JULIA_CONFIG := $(JULIA) -e 'include(joinpath(Sys.BINDIR, Base.DATAROOTDIR, "julia", "julia-config.jl"))' --
JULIA_LIBDIR := $(shell $(JULIA) --startup-file=no -e 'println(joinpath(Sys.BINDIR, "..", "lib"))' --)
JULIA_LIBDIR := $(shell $(JULIA) --startup-file=no -e 'println(joinpath(Sys.BINDIR, Base.LIBDIR))' --)
CPPFLAGS_ADD :=
CFLAGS_ADD = $(shell $(JULIA_CONFIG) --cflags)
LDFLAGS_ADD = -lm $(shell $(JULIA_CONFIG) --ldflags --ldlibs) -ljulia-internal
Expand Down