Move parallel precompilation to Base #53403
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KristofferC
added
domain:packages
KristofferC
force-pushed
the
kc/base_precompile
branch
2 times, most recently
from
1666dc0
to
58e3580
Compare
https://github.com/JuliaLang/Downloads.jl/blob/2dd891ab237ad6a58b11337027325fb571f43cb7/src/Downloads.jl#L469-L474 is buggy and will not be relocatable since |
KristofferC
force-pushed
the
kc/base_precompile
branch
from
77b4e4d
to
edcf4a6
Compare
KristofferC
force-pushed
the
kc/base_precompile
branch
from
4a04df3
to
bf05d28
Compare
KristofferC
force-pushed
the
kc/base_precompile
branch
2 times, most recently
from
866cc1c
to
bf196e4
Compare
KristofferC
added a commit
that referenced
this pull request
Mar 1, 2024
Backported PRs: - [x] #53361 <!-- 🤖 [master] Bump the SparseArrays stdlib from c9f7293 to cb602d7 --> - [x] #53300 <!-- allow external absint to hold custom data in `codeinst.inferred` --> - [x] #53342 <!-- Add `Base.wrap` to docs --> - [x] #53372 <!-- Silence warnings in `test/file.jl` --> - [x] #53357 <!-- 🤖 [master] Bump the Pkg stdlib from 6dd0e7c9e to 76070d295 --> - [x] #53373 <!-- fix sysimage-native-code=no option with pkgimages --> - [x] #53333 <!-- More consistent return value for annotations, take 2 --> - [x] #53354 <!-- fix code coverage bug in tail position and `else` --> - [x] #53407 <!-- fix sysimage-native-code=yes option --> - [x] #53388 <!-- Fix documentation: thread pool of main thread --> - [x] #53355 <!-- Fix synchronization issues on the GC scheduler. --> - [x] #53429 <!-- Subtype: skip slow-path in `local_∀_∃_subtype` if inputs contain no ∃ typevar. --> - [x] #53437 <!-- Add debug variant of loader_trampolines.o --> - [x] #53284 <!-- Add annotate! method for AnnotatedIOBuffer --> - [x] #53466 <!-- [MozillaCACerts_jll] Update to v2023-12-12 --> - [x] #53467 <!-- [LibGit2_jll] Update to v1.7.2 --> - [x] #53326 <!-- RFC: when loading code for internal purposes, load stdlib files directly, bypassing DEPOT_PATH, LOAD_PATH, and stale checks --> - [x] #53332 - [x] #53320 <!-- Add `Sys.isreadable, Sys.iswriteable`, update `ispath` --> - [x] #53476 Contains multiple commits, manual intervention needed: - [ ] #53285 <!-- Add update mechanism for Terminfo, and common user-alias data --> Non-merged PRs with backport label: - [ ] #53424 <!-- yet more atomics & cache-line fixes on work-stealing queue --> - [ ] #53408 <!-- task splitting: change additive accumulation to multiplicative --> - [ ] #53403 <!-- Move parallel precompilation to Base --> - [ ] #53402 <!-- Add `jl_getaffinity` and `jl_setaffinity` --> - [ ] #53391 <!-- Default to the medium code model in x86 linux --> - [ ] #53125 <!-- coverage: count coverage where explicitly requested by inference only --> - [ ] #52694 <!-- Reinstate similar for AbstractQ for backward compatibility -->
60 tasks
This could be used in the future to implement code loading on top of as well
KristofferC
force-pushed
the
kc/base_precompile
branch
2 times, most recently
from
894439e
to
f316bf8
Compare
KristofferC
force-pushed
the
kc/base_precompile
branch
from
f316bf8
to
3417484
Compare
IanButterworth
approved these changes
Mar 4, 2024
There was a problem hiding this comment.
I've renamed the module PrecompilePkgs->Precompilation and file to precompilation.jl to match the form of loading.jl.
I've also made non-interactive printing always show with timing. It is probably helpful to do-so for CI.
Note that loading-invoked precompilation in a noninteractive session is still silent, like it is for serial precompile in julia currently.
Thank you @KristofferC for making this happen.
IanButterworth
added
the
status:merge me
IanButterworth
removed
the
status:merge me
1 task
KristofferC
added a commit
that referenced
this pull request
Mar 6, 2024
Parallel precompilation is more or less now required in order to use somewhat large packages unless you want to wait an obscene long time for it to complete. Right now, we even start a parallel precompilation on a package load if we notice that the package you are loading is not precompiled. This functionally has typically been implemented in Pkg but with Pkg not being in the sysimage it becomes a bit awkward because we then need to load Pkg from Base. The only real reason this functionality has been implemented in Pkg is that Pkg has some useful features for parsing environments. Moving precompilation to Base has typically been stalled on such an environment parser not existing in Base. However, in #46690 I started implemented code loading on top of a more up front environment parser (instead of the "incremental" one that currently exists in `loading.jl`) and we can retro fit this to be used as the basis of parallel precompilation. At some later point code loading could be implemented on top of it but that is for now considered future work. This PR thus adds the environment parser from the codeloading PR and implementes the parallel precompilation feature from Pkg on top of it (instead of on top of the `EnvCache` in Pkg). Some points to bring up here: - This copy pastes the progress bar implementation in Pkg into here. It is probably a bit excessive to use so we can simplify that significantly. - Parallel precompilation uses the `FileWatching` module to avoid different processes trying to precompile the same package concurrently. Right now, I used grab this from `Base.loaded_modules` relying on it being in the sysimage. - This removes the "suspended" functionality from the Pkg precompilation which does not try to precompile packages if they have "recently" failed which is unclear how useful it is in practice. This also requires the Serialization stdlib and uses data structures defined in Pkg so it is hard to keep when moving this to Base. --------- Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com> (cherry picked from commit 6745160)
vchuravy
added a commit
that referenced
this pull request
Mar 8, 2024
Follow-on from #53403 This extends `Base.Precompilation.precompilepkgs` to take a list of configurations to precompile each package with, while parallelizing across all packages and configurations, and uses it to build the stdlib pkgimages. It simplifies the stdlib pkgimage build process but is (currently) dependent on having an accurately resolved Manifest.toml (Project.toml included to make the manifest easier to make). Any new/removed stdlibs or changes their dependencies will require updating the Manifest.toml. It's a bit chicken and egg, but should be manageable with manual editing of the Manifest.toml. In terms of speed improvement: MacOS aarch64 CI runner 6m19s before, 5m19 with this Note that CI builds will show the basic print with timing of each package, whereas local build will be the tidier fancy print without timings. Co-authored-by: Valentin Churavy <vchuravy@users.noreply.github.com>
KristofferC
pushed a commit
that referenced
this pull request
Mar 15, 2024
Follow-on from #53403 This extends `Base.Precompilation.precompilepkgs` to take a list of configurations to precompile each package with, while parallelizing across all packages and configurations, and uses it to build the stdlib pkgimages. It simplifies the stdlib pkgimage build process but is (currently) dependent on having an accurately resolved Manifest.toml (Project.toml included to make the manifest easier to make). Any new/removed stdlibs or changes their dependencies will require updating the Manifest.toml. It's a bit chicken and egg, but should be manageable with manual editing of the Manifest.toml. In terms of speed improvement: MacOS aarch64 CI runner 6m19s before, 5m19 with this Note that CI builds will show the basic print with timing of each package, whereas local build will be the tidier fancy print without timings. Co-authored-by: Valentin Churavy <vchuravy@users.noreply.github.com> (cherry picked from commit 78351b5)
KristofferC
added a commit
that referenced
this pull request
Mar 17, 2024
Backported PRs: - [x] #39071 <!-- Add a lazy `logrange` function and `LogRange` type --> - [x] #51802 <!-- Allow AnnotatedStrings in log messages --> - [x] #53369 <!-- Orthogonalize re-indexing for FastSubArrays --> - [x] #48050 <!-- improve `--heap-size-hint` arg handling --> - [x] #53482 <!-- add IR encoding for EnterNode --> - [x] #53499 <!-- Avoid compiler warning about redefining jl_globalref_t --> - [x] #53507 <!-- update staled `Core.Compiler.Effects` documentation --> - [x] #53408 <!-- task splitting: change additive accumulation to multiplicative --> - [x] #53523 <!-- add back an alias for `check_top_bit` --> - [x] #53377 <!-- add _readdirx for returning more object info gathered during dir scan --> - [x] #53525 <!-- fix InteractiveUtils call in Base.runtests on failure --> - [x] #53540 <!-- use more efficient `_readdirx` for tab completion --> - [x] #53545 <!-- use `_readdirx` for `walkdir` --> - [x] #53551 <!-- revert "Add @create_log_macro for making custom styled logging macros (#52196)" --> - [x] #53554 <!-- Always return a value in 1-d circshift! of abstractarray.jl --> - [x] #53424 <!-- yet more atomics & cache-line fixes on work-stealing queue --> - [x] #53571 <!-- Update Documenter to v1.3 for inventory writing --> - [x] #53403 <!-- Move parallel precompilation to Base --> - [x] #53589 <!-- add back `unsafe_convert` to pointer for arrays --> - [x] #53596 <!-- build: remove extra .a file --> - [x] #53606 <!-- fix error path in `precompilepkgs` --> - [x] #53004 <!-- Unexport with, at_with, and ScopedValue from Base --> - [x] #53629 <!-- typo fix in scoped values docs --> - [x] #53630 <!-- sroa: Fix incorrect scope counting --> - [x] #53598 <!-- Use Base parallel precompilation to build stdlibs --> - [x] #53649 <!-- precompilepkgs: package in boths deps and weakdeps are in fact only weak --> - [x] #53671 <!-- Fix bootstrap Base precompile in cross compile configuration --> - [x] #52125 <!-- Load Pkg if not already to reinstate missing package add prompt --> - [x] #53602 <!-- Handle zero on arrays of unions of number types and missings --> - [x] #53516 <!-- permit NamedTuple{<:Any, Union{}} to be created --> - [x] #53643 <!-- Bump CSL to 1.1.1 to fix libgomp bug --> - [x] #53679 <!-- move precompile workload back from Base --> - [x] #53663 <!-- add isassigned methods for reinterpretarray --> - [x] #53662 <!-- [REPL] fix incorrectly cleared line after completions accepted --> - [x] #53611 <!-- Linalg: matprod_dest for Diagonal and adjvec --> - [x] #53659 <!-- fix #52025, re-allow all implicit pointer casts in cconvert for Array --> - [x] #53631 <!-- LAPACK: validate input parameters to throw informative errors --> - [x] #53628 <!-- Make some improvements to the Scoped Values documentation. --> - [x] #53655 <!-- Change tbaa of ptr_phi to tbaa_value --> - [x] #53391 <!-- Default to the medium code model in x86 linux --> - [x] #53699 <!-- Move `isexecutable, isreadable, iswritable` to `filesystem.jl` --> - [x] #41232 <!-- Fix linear indexing for ReshapedArray if the parent has offset axes --> - [x] #53527 <!-- Enable analyzegc checks for try catch and fix found issues --> - [x] #52092 - [x] #53682 <!-- Increase build precompilation --> - [x] #53720 - [x] #53553 <!-- typeintersect: fix `UnionAll` unaliasing bug caused by innervars. --> Contains multiple commits, manual intervention needed: - [ ] #53305 <!-- Propagate inbounds in isassigned with CartesianIndex indices --> Non-merged PRs with backport label: - [ ] #53736 <!-- fix literal-pow to return the right type when the base is -1 --> - [ ] #53707 <!-- Make ScopedValue public --> - [ ] #53696 <!-- add invokelatest to on_done callback in bracketed paste --> - [ ] #53660 <!-- put Logging back in default sysimage --> - [ ] #53509 <!-- revert moving "creating packages" from Pkg.jl --> - [ ] #53452 <!-- RFC: allow Tuple{Union{}}, returning Union{} --> - [ ] #53402 <!-- Add `jl_getaffinity` and `jl_setaffinity` --> - [ ] #52694 <!-- Reinstate similar for AbstractQ for backward compatibility --> - [ ] #51928 <!-- Styled markdown, with a few tweaks --> - [ ] #51816 <!-- User-themable stacktraces --> - [ ] #51811 <!-- Make banner size depend on terminal size --> - [ ] #51479 <!-- prevent code loading from lookin in the versioned environment when building Julia -->
KristofferC
removed
the
backport 1.11
mkitti
pushed a commit
to mkitti/julia
that referenced
this pull request
Apr 13, 2024
Parallel precompilation is more or less now required in order to use somewhat large packages unless you want to wait an obscene long time for it to complete. Right now, we even start a parallel precompilation on a package load if we notice that the package you are loading is not precompiled. This functionally has typically been implemented in Pkg but with Pkg not being in the sysimage it becomes a bit awkward because we then need to load Pkg from Base. The only real reason this functionality has been implemented in Pkg is that Pkg has some useful features for parsing environments. Moving precompilation to Base has typically been stalled on such an environment parser not existing in Base. However, in JuliaLang#46690 I started implemented code loading on top of a more up front environment parser (instead of the "incremental" one that currently exists in `loading.jl`) and we can retro fit this to be used as the basis of parallel precompilation. At some later point code loading could be implemented on top of it but that is for now considered future work. This PR thus adds the environment parser from the codeloading PR and implementes the parallel precompilation feature from Pkg on top of it (instead of on top of the `EnvCache` in Pkg). Some points to bring up here: - This copy pastes the progress bar implementation in Pkg into here. It is probably a bit excessive to use so we can simplify that significantly. - Parallel precompilation uses the `FileWatching` module to avoid different processes trying to precompile the same package concurrently. Right now, I used grab this from `Base.loaded_modules` relying on it being in the sysimage. - This removes the "suspended" functionality from the Pkg precompilation which does not try to precompile packages if they have "recently" failed which is unclear how useful it is in practice. This also requires the Serialization stdlib and uses data structures defined in Pkg so it is hard to keep when moving this to Base. --------- Co-authored-by: Ian Butterworth <i.r.butterworth@gmail.com>
mkitti
pushed a commit
to mkitti/julia
that referenced
this pull request
Apr 13, 2024
Follow-on from JuliaLang#53403 This extends `Base.Precompilation.precompilepkgs` to take a list of configurations to precompile each package with, while parallelizing across all packages and configurations, and uses it to build the stdlib pkgimages. It simplifies the stdlib pkgimage build process but is (currently) dependent on having an accurately resolved Manifest.toml (Project.toml included to make the manifest easier to make). Any new/removed stdlibs or changes their dependencies will require updating the Manifest.toml. It's a bit chicken and egg, but should be manageable with manual editing of the Manifest.toml. In terms of speed improvement: MacOS aarch64 CI runner 6m19s before, 5m19 with this Note that CI builds will show the basic print with timing of each package, whereas local build will be the tidier fancy print without timings. Co-authored-by: Valentin Churavy <vchuravy@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Parallel precompilation is more or less now required in order to use somewhat large packages unless you want to wait an obscene long time for it to complete. Right now, we even start a parallel precompilation on a package load if we notice that the package you are loading is not precompiled.
This functionally has typically been implemented in Pkg but with Pkg not being in the sysimage it becomes a bit awkward because we then need to load Pkg from Base. The only real reason this functionality has been implemented in Pkg is that Pkg has some useful features for parsing environments. Moving precompilation to Base has typically been stalled on such an environment parser not existing in Base.
However, in #46690 I started implemented code loading on top of a more up front environment parser (instead of the "incremental" one that currently exists in
loading.jl) and we can retro fit this to be used as the basis of parallel precompilation. At some later point code loading could be implemented on top of it but that is for now considered future work.This PR thus adds the environment parser from the codeloading PR and implementes the parallel precompilation feature from Pkg on top of it (instead of on top of the
EnvCachein Pkg).Some points to bring up here:
FileWatchingmodule to avoid different processes trying to precompile the same package concurrently. Right now, I used grab this fromBase.loaded_modulesrelying on it being in the sysimage.