Precompile Pkg.activate() #3540
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I enhanced this now activate all the pakcages in |
|
With these changes with Julia master, I get the following results. |
|
One compilation I noticed is addressed via JuliaLang/julia#50416 |
|
I don't see any compilation after the first |
|
Here's what I saw from Julia 1.9.0: |
|
Probably doesn't matter but 1.9.1 is the latest version so always use that. Also, remove From what I see, this PR calls |
|
Here is Julia 1.9.2 (release-1.9 branch) without |
|
I ran the following excerpt with Julia 1.9.2: test.jl: `$ time ./julia --trace-compile=stderr --startup-file=no test.jl` |
|
I could probably be more selective about the test package list if you would like. Perhaps something like this based on Julia 1.9.2 results? |
|
On Julia master: `$ time ./julia --trace-compile=stderr --startup-file=no test.jl` with Julia Version 1.11.0-DEV.14 |
|
I narrowed down the list. Here are the |
|
should we merge this? |
|
Do we really need to check in Manifests? |
|
We already have several manifests in test/test_packages. I can check whether it is needed this weekend. I'm currently traveling. |
mkitti
force-pushed
the
mkitti/precompile_activate
branch
from
aa1e79c
to
b47db96
Compare
|
There are 50+ packages in |
|
I had previously selected several packages to target in Julia 1.9, but I'm not sure if those packages are still the optimal set for Julia master. I'm rather confused on the state of precompilation though now that this is not in the system image. |
#!./julia --trace-compile=stderr
import Pkg
function main()
for test_packages_path in (
abspath(joinpath(dirname(pathof(Pkg)), ".." ,"test", "test_packages")),
joinpath(ENV["HOME"], ".julia/dev")
)
for i in 1:3
@info "Starting activation run" test_packages_path
@time for test_package in readdir(test_packages_path)
Pkg.activate(joinpath(test_packages_path, test_package))
end
@info "Ending activation run"
end
end
end
main()Here's what I see on the initial pass: precompile(Tuple{typeof(Base.repeat), Char, Int64})
Activating project at `@stdlib/A`
precompile(Tuple{typeof(Base.convert), Type{Base.Dict{String, Array{String, 1}}}, Base.Dict{String, Any}})
precompile(Tuple{typeof(Base.TOML.try_return_datetime), Base.TOML.Parser, Vararg{Int64, 7}})
precompile(Tuple{Type{Dates.DateTime}, Vararg{Int64, 7}})
precompile(Tuple{Type{Base.Generator{I, F} where F where I}, Pkg.Types.var"#52#55"{String, String}, Array{Any, 1}})
precompile(Tuple{typeof(Base.collect_similar), Array{Any, 1}, Base.Generator{Array{Any, 1}, Pkg.Types.var"#52#55"{String, String}}})
precompile(Tuple{Pkg.Types.var"#52#55"{String, String}, Base.Dict{String, Any}})
precompile(Tuple{Type{Array{Dates.DateTime, 1}}, UndefInitializer, Tuple{Int64}})
precompile(Tuple{typeof(Base.collect_to_with_first!), Array{Dates.DateTime, 1}, Dates.DateTime, Base.Generator{Array{Any, 1}, Pkg.Types.var"#52#55"{String, String}}, Int64})
precompile(Tuple{typeof(Base.reduce_empty), Base.MappingRF{typeof(Base.identity), typeof(Base.max)}, Type{Dates.DateTime}})
precompile(Tuple{typeof(Base.maximum), Array{Dates.DateTime, 1}})
precompile(Tuple{Type{Pair{A, B} where B where A}, String, Dates.DateTime})
precompile(Tuple{typeof(Base.map), Function, Array{Base.Dict{String, Dates.DateTime}, 1}})
precompile(Tuple{typeof(TOML.Internals.Printer.is_array_of_tables), Array{Base.Dict{String, Dates.DateTime}, 1}})
precompile(Tuple{typeof(Core.kwcall), NamedTuple{(:indent, :sorted, :by), Tuple{Int64, Bool, typeof(Base.identity)}}, typeof(Base.invokelatest), Any, Any, Vararg{Any}})
precompile(Tuple{Base.var"##invokelatest#2", Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:indent, :sorted, :by), Tuple{Int64, Bool, typeof(Base.identity)}}}, typeof(Base.invokelatest), Any, Any, Vararg{Any}})
precompile(Tuple{typeof(Core.kwcall), NamedTuple{(:indent, :sorted, :by), Tuple{Int64, Bool, typeof(Base.identity)}}, typeof(TOML.Internals.Printer.print_table), Nothing, Base.IOStream, Base.Dict{String, Dates.DateTime}, Array{String, 1}})
precompile(Tuple{typeof(TOML.Internals.Printer.is_array_of_tables), Dates.DateTime})
precompile(Tuple{Base.var"#774#775"{FileWatching.Pidfile.var"#2#4"{Base.Filesystem.File}, Base.Timer}})
precompile(Tuple{typeof(Base.similar), Array{Any, 1}})
precompile(Tuple{typeof(Base.Iterators.enumerate), Array{Any, 1}})
precompile(Tuple{typeof(Base.similar), Array{String, 1}})
precompile(Tuple{typeof(Base.Iterators.enumerate), Array{String, 1}})
precompile(Tuple{typeof(Base.setindex!), Array{String, 1}, String, Int64})
precompile(Tuple{typeof(Core.Compiler.eltype), Type{Array{Union{}, 1}}})
precompile(Tuple{typeof(Base.deepcopy_internal), Tuple{}, Base.IdDict{Any, Any}})
precompile(Tuple{typeof(Base.deepcopy_internal), Base.Dict{String, Base.UUID}, Base.IdDict{Any, Any}})
precompile(Tuple{typeof(Base.deepcopy_internal), Base.Dict{String, Union{Array{String, 1}, String}}, Base.IdDict{Any, Any}})
precompile(Tuple{typeof(Base.deepcopy_internal), Base.Dict{String, Array{String, 1}}, Base.IdDict{Any, Any}})
precompile(Tuple{Type{GenericMemory{:not_atomic, String, Core.AddrSpace{Core}(0x00)}}, UndefInitializer, Int64})
Activating project at `@stdlib/ActiveProjectInTestSubgraph`
Activating project at `@stdlib/ArtifactInstallation`
Activating project at `@stdlib/ArtifactOverrideLoading`
Activating new project at `@stdlib/ArtifactTOMLSearch`
Activating project at `@stdlib/AugmentedPlatform`
Activating project at `@stdlib/BasicCompat`
precompile(Tuple{typeof(Base.getproperty), Base.MappingRF{Base.var"#341#342"{Pkg.Versions.var"#4#5"}, Base.BottomRF{typeof(Base.add_sum)}}, Symbol})
precompile(Tuple{Pkg.Versions.var"#4#5", Nothing})
precompile(Tuple{Base.MappingRF{Base.var"#341#342"{Pkg.Versions.var"#4#5"}, Base.BottomRF{typeof(Base.add_sum)}}, Int64, Nothing})
precompile(Tuple{Type{Pkg.Versions.VersionBound}, Tuple{Int64}})
precompile(Tuple{Type{Pkg.Versions.VersionBound}, Tuple{Int64, Int64}})
precompile(Tuple{typeof(Pkg.Versions.semver_interval), Base.RegexMatch{String}})
precompile(Tuple{typeof(Base.push!), Array{Pkg.Versions.VersionRange, 1}, Pkg.Versions.VersionRange})
precompile(Tuple{typeof(Base.getproperty), Base.MappingRF{Base.var"#341#342"{Pkg.Versions.var"#6#7"}, Base.BottomRF{typeof(Base.add_sum)}}, Symbol})
precompile(Tuple{Pkg.Versions.var"#6#7", Nothing})
precompile(Tuple{Base.MappingRF{Base.var"#341#342"{Pkg.Versions.var"#6#7"}, Base.BottomRF{typeof(Base.add_sum)}}, Int64, Nothing})
precompile(Tuple{Type{Pkg.Versions.VersionBound}, Int64, Vararg{Int64}})
precompile(Tuple{typeof(Pkg.Versions.inequality_interval), Base.RegexMatch{String}})
Activating project at `@stdlib/BasicSandbox`
# ...
Activating project at `@stdlib/CompatOutOfSync`
precompile(Tuple{typeof(Pkg.Types.read_project_uuid), Nothing})
Activating project at `@stdlib/DependsOnExample`
# ...
Activating project at `@stdlib/NotUpdated`
precompile(Tuple{typeof(Base.deepcopy_internal), Tuple{String}, Base.IdDict{Any, Any}})
Activating project at `@stdlib/PackageWithDependency`
Activating project at `@stdlib/RecurringPrecompile`
precompile(Tuple{typeof(Core.Compiler.eltype), Type{Array{UInt64, 1}}})
precompile(Tuple{typeof(Base.deepcopy_internal), Tuple{UInt64}, Base.IdDict{Any, Any}})
Activating project at `@stdlib/SameNameDifferentUUID`I'm not sure how much of that is cacheable in a pkgimage, but maybe this will be useful if this functionality is moved to |
|
On the second pass I see |
|
I removed extra packages and manifests. This is pretty straightforward now. @oscardssmith merge? |
Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com>
IanButterworth
approved these changes
Nov 1, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pkg.activateis a necessary first step for running executable Julia scripts.For a novice Julia user, it is often easier to provide them with a script to run simply via
julia script.jl.script.jl may contain the following contents.
While trying to build such scripts, I found that
Pkg.activateis contributing some latency.Activating the environment at https://github.com/mkitti/ScriptUtils.jl/tree/main/examples/Echo.jl
can take 100 to 200 milliseconds on my computer, with most of the time being compilation. Even a
second activation requires an additional 40 mlliseconds of compilation time.
Alternating between the default environment and the current directory, also seems to involve compilation
up to four activations.
As such I propose adding four
Pkg.activatecalls to the Pkg.jl precompilation script, alternatingbetween the default environment and TestPkg.jl.