Backports for 1.10.3 #53714
Merged
Backports for 1.10.3 #53714
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Keno Fischer <keno@juliahub.com> (cherry picked from commit b8540d1)
(cherry picked from commit 71f68b4)
This regression at a large number of GC threads still shows up on a large aarch64 machine (80-core ARM Neoverse N1). ``` bench = "many_refs.jl" (gcthreads, threads) = (1, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 3807 │ 2826 │ 2633 │ 193 │ 960 │ 13 │ 892 │ 74 │ │ median │ 3810 │ 2826 │ 2633 │ 193 │ 961 │ 15 │ 892 │ 74 │ │ maximum │ 3810 │ 2831 │ 2638 │ 193 │ 962 │ 27 │ 892 │ 74 │ │ stdev │ 2 │ 3 │ 3 │ 0 │ 1 │ 7 │ 0 │ 0 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (2, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2455 │ 1476 │ 1350 │ 126 │ 488 │ 20 │ 892 │ 60 │ │ median │ 2459 │ 1478 │ 1352 │ 126 │ 489 │ 23 │ 892 │ 60 │ │ maximum │ 2465 │ 1479 │ 1352 │ 126 │ 489 │ 23 │ 893 │ 60 │ │ stdev │ 5 │ 1 │ 1 │ 0 │ 1 │ 2 │ 0 │ 0 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (4, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2173 │ 1194 │ 1115 │ 77 │ 463 │ 18 │ 892 │ 54 │ │ median │ 2187 │ 1194 │ 1116 │ 78 │ 463 │ 19 │ 893 │ 55 │ │ maximum │ 2217 │ 1208 │ 1130 │ 78 │ 463 │ 19 │ 893 │ 55 │ │ stdev │ 22 │ 8 │ 8 │ 1 │ 0 │ 1 │ 0 │ 0 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (8, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2191 │ 1212 │ 1149 │ 63 │ 539 │ 19 │ 901 │ 55 │ │ median │ 2199 │ 1219 │ 1156 │ 63 │ 540 │ 20 │ 901 │ 55 │ │ maximum │ 2201 │ 1222 │ 1157 │ 65 │ 540 │ 20 │ 901 │ 56 │ │ stdev │ 5 │ 5 │ 4 │ 1 │ 1 │ 1 │ 0 │ 0 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (16, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 3897 │ 2916 │ 2854 │ 60 │ 1423 │ 20 │ 902 │ 75 │ │ median │ 4012 │ 3032 │ 2972 │ 62 │ 1491 │ 21 │ 904 │ 75 │ │ maximum │ 4026 │ 3106 │ 3044 │ 62 │ 1511 │ 22 │ 904 │ 77 │ │ stdev │ 70 │ 96 │ 96 │ 1 │ 46 │ 1 │ 1 │ 1 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ ``` This patch seems to improve performance at 16 GC threads though we still have some negative scaling for this benchmark beyond 8 GC threads (others seem fine). I didn't test whether this is indeed the optimal struct layout for the queue in this benchmark. ``` bench = "many_refs.jl" (gcthreads, threads) = (1, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 3807 │ 2822 │ 2630 │ 193 │ 959 │ 15 │ 892 │ 74 │ │ median │ 3824 │ 2823 │ 2630 │ 193 │ 959 │ 15 │ 892 │ 74 │ │ maximum │ 4307 │ 3194 │ 2974 │ 220 │ 1086 │ 19 │ 892 │ 74 │ │ stdev │ 284 │ 215 │ 199 │ 16 │ 74 │ 2 │ 0 │ 0 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (2, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2459 │ 1476 │ 1349 │ 126 │ 488 │ 19 │ 892 │ 60 │ │ median │ 2460 │ 1479 │ 1352 │ 127 │ 488 │ 21 │ 893 │ 60 │ │ maximum │ 2770 │ 1661 │ 1526 │ 135 │ 570 │ 22 │ 893 │ 60 │ │ stdev │ 179 │ 106 │ 101 │ 5 │ 47 │ 1 │ 0 │ 0 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (4, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2215 │ 1233 │ 1155 │ 77 │ 485 │ 20 │ 892 │ 53 │ │ median │ 2320 │ 1248 │ 1171 │ 78 │ 492 │ 21 │ 893 │ 54 │ │ maximum │ 2372 │ 1259 │ 1176 │ 83 │ 493 │ 22 │ 893 │ 56 │ │ stdev │ 80 │ 13 │ 11 │ 3 │ 5 │ 1 │ 1 │ 1 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (8, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2204 │ 1220 │ 1156 │ 64 │ 544 │ 23 │ 901 │ 53 │ │ median │ 2362 │ 1249 │ 1181 │ 64 │ 553 │ 23 │ 901 │ 53 │ │ maximum │ 2374 │ 1261 │ 1196 │ 68 │ 558 │ 25 │ 901 │ 55 │ │ stdev │ 95 │ 21 │ 20 │ 2 │ 7 │ 1 │ 0 │ 1 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ (gcthreads, threads) = (16, 1) [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback [ Info: Setting GC memory pressure callback ┌─────────┬────────────┬─────────┬───────────┬────────────┬──────────────┬───────────────────┬──────────┬────────────┐ │ │ total time │ gc time │ mark time │ sweep time │ max GC pause │ time to safepoint │ max heap │ percent gc │ │ │ ms │ ms │ ms │ ms │ ms │ us │ MB │ % │ ├─────────┼────────────┼─────────┼───────────┼────────────┼──────────────┼───────────────────┼──────────┼────────────┤ │ minimum │ 2502 │ 1519 │ 1458 │ 62 │ 721 │ 22 │ 902 │ 58 │ │ median │ 2511 │ 1524 │ 1461 │ 63 │ 728 │ 23 │ 903 │ 61 │ │ maximum │ 2664 │ 1554 │ 1486 │ 68 │ 741 │ 25 │ 905 │ 61 │ │ stdev │ 91 │ 19 │ 16 │ 3 │ 10 │ 1 │ 2 │ 1 │ └─────────┴────────────┴─────────┴───────────┴────────────┴──────────────┴───────────────────┴──────────┴────────────┘ ``` (cherry picked from commit 973b4ff)
(cherry picked from commit e9c84c8)
Cherry-pick the parts of #53355 which are relevant to the 1.10 GC.
This shouldn't have any cost on smaller images because the only thing that gets put into ldata is the system image data, which is only reference via `dlsym`. This allows for images larger than 2gb (tested by putting a 2gb array in the base image) I did not test how this might be handled in other platforms (Windows doesn't support it). (cherry picked from commit 0f04b33)
(cherry picked from commit abd00d0)
~~Apparently somewhere in codegen inside `emit_codeinst`some piece of code is relying on the implicit try catch gcstack restoring. I haven't got the analyzer working on that file yet (it has hundreds of issues and it doesn't like C++ code that much + the file is tens of thousands of lines after includes so it struggles).~~ This fixes the compileall segfault in apple-aarch64 ci. #53811 (cherry picked from commit 52fc796)
…41232) This PR fixes ```julia julia> r = reshape(Base.IdentityUnitRange(3:4), 2, 1) 2×1 reshape(::Base.IdentityUnitRange{UnitRange{Int64}}, 2, 1) with eltype Int64: 3 4 julia> collect(r) == r false julia> collect(r) 2×1 Matrix{Int64}: 3258125826116431922 3688512103538242609 ``` After this PR, ```julia julia> collect(r) 2×1 Matrix{Int64}: 3 4 ``` (cherry picked from commit 6e3044d)
|
It'd be great to get 1.10.3 out (with the threading gc segfault fix that's on this branch) |
Closes #54015. `diff -r` on the source tree for the original archive vs. the archive that gets downloaded now shows no changes.
This reduces dynamic dispatch and makes JET happier.
Testing on v1.11:
```julia
julia> import LinearAlgebra: checksquare
julia> using JET
julia> @report_opt checksquare(rand(2,2), rand(2,2))
═════ 4 possible errors found ═════
┌ checksquare(::Matrix{Float64}, ::Matrix{Float64}) @ LinearAlgebra /cache/build/builder-amdci4-1/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/LinearAlgebra/src/LinearAlgebra.jl:307
│┌ string(::String, ::Tuple{Int64, Int64}) @ Base ./strings/io.jl:189
││┌ print_to_string(::String, ::Tuple{Int64, Int64}) @ Base ./strings/io.jl:150
│││┌ _unsafe_take!(io::IOBuffer) @ Base ./iobuffer.jl:494
││││┌ wrap(::Type{Array}, m::MemoryRef{UInt8}, l::Int64) @ Base ./array.jl:3101
│││││ failed to optimize due to recursion: wrap(::Type{Array}, ::MemoryRef{UInt8}, ::Int64)
││││└────────────────────
│││┌ print_to_string(::String, ::Vararg{Any}) @ Base ./strings/io.jl:143
││││ runtime dispatch detected: Base._str_sizehint(%17::Any)::Int64
│││└────────────────────
│││┌ print_to_string(::String, ::Vararg{Any}) @ Base ./strings/io.jl:148
││││ runtime dispatch detected: print(%59::IOBuffer, %97::Any)::Any
│││└────────────────────
│││┌ string(::String, ::Int64, ::String, ::Tuple{Int64}, ::String, ::Int64, ::String, ::Int64, ::String) @ Base ./strings/io.jl:189
││││ failed to optimize due to recursion: string(::String, ::Int64, ::String, ::Tuple{Int64}, ::String, ::Int64, ::String, ::Int64, ::String)
│││└────────────────────
julia> function checksquare(A...) # This PR
sizes = Int[]
for a in A
size(a,1)==size(a,2) || throw(DimensionMismatch(lazy"matrix is not square: dimensions are $(size(a))"))
push!(sizes, size(a,1))
end
return sizes
end
checksquare (generic function with 2 methods)
julia> @report_opt checksquare(rand(2,2), rand(2,2))
No errors detected
```
(cherry picked from commit d505c8c)
This was referenced Apr 11, 2024
…626b0 (#54049) Stdlib: Pkg URL: https://github.com/JuliaLang/Pkg.jl.git Stdlib branch: release-1.10 Julia branch: backports-release-1.10 Old commit: 70525539d New commit: f487626b0 Julia version: 1.10.2 Pkg version: 1.10.0(Does not match) Bump invoked by: @IanButterworth Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: JuliaLang/Pkg.jl@7052553...f487626 ``` $ git log --oneline 70525539d..f487626b0 f487626b0 Merge pull request #3868 from JuliaLang/backports-release-1.10 e4a6078a2 prune manifest after the set of some deps have been "demoted" to weakdeps (#3864) 72ba57759 Report failures to download artifacts as failures (#3860) 4ee732f1b why: show more when package is both a direct and indirect dep (#3771) ``` Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
IanButterworth
force-pushed
the
backports-release-1.10
branch
from
d561ec2
to
e386554
Compare
|
feceefe appears to have introduced a LinearAlgebra failure https://buildkite.com/julialang/julia-release-1-dot-10/builds/245#018ecd32-81ec-4189-99ed-8fcbb37d3b6c/634-688 which seems odd |
#54063) Stdlib: SHA URL: https://github.com/JuliaCrypto/SHA.jl.git Stdlib branch: release-0.7 Julia branch: backports-release-1.10 Old commit: 2d1f84e New commit: e1af7dd Julia version: 1.10.2 SHA version: 0.7.0(Does not match) Bump invoked by: @IanButterworth Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: JuliaCrypto/SHA.jl@2d1f84e...e1af7dd ``` $ git log --oneline 2d1f84e..e1af7dd e1af7dd Hardcode doc edit backlink aaf2df6 [test] fix typo 0f70e8b [test] test `AbstractString` input 335322a [test] test branch in `digest!` 5992826 [test] Testing branch within the `HMAC_CTX` constructor 5e3a676 Protect against re-using digested context ``` Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
…5 to 8eec5cb (#54065) Stdlib: NetworkOptions URL: https://github.com/JuliaLang/NetworkOptions.jl.git Stdlib branch: master Julia branch: backports-release-1.10 Old commit: 0bd3345 New commit: 8eec5cb Julia version: 1.10.2 NetworkOptions version: 1.2.0(Does not match) Bump invoked by: @IanButterworth Powered by: [BumpStdlibs.jl](https://github.com/JuliaLang/BumpStdlibs.jl) Diff: JuliaLang/NetworkOptions.jl@0bd3345...8eec5cb ``` $ git log --oneline 0bd3345..8eec5cb 8eec5cb Hardcode doc edit backlink (#32) ``` Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
IanButterworth
force-pushed
the
backports-release-1.10
branch
from
1e28984
to
97b0571
Compare
|
I confirmed locally that this error was introduced by which is weird for just a change switching to lazy strings? diff --git a/stdlib/LinearAlgebra/src/LinearAlgebra.jl b/stdlib/LinearAlgebra/src/LinearAlgebra.jl
index 494572b1e8dd8..7b3ff8f0db53d 100644
--- a/stdlib/LinearAlgebra/src/LinearAlgebra.jl
+++ b/stdlib/LinearAlgebra/src/LinearAlgebra.jl
@@ -238,14 +238,14 @@ julia> LinearAlgebra.checksquare(A, B)
"""
function checksquare(A)
m,n = size(A)
- m == n || throw(DimensionMismatch("matrix is not square: dimensions are $(size(A))"))
+ m == n || throw(DimensionMismatch(lazy"matrix is not square: dimensions are $(size(A))"))
m
end
function checksquare(A...)
sizes = Int[]
for a in A
- size(a,1)==size(a,2) || throw(DimensionMismatch("matrix is not square: dimensions are $(size(a))"))
+ size(a,1)==size(a,2) || throw(DimensionMismatch(lazy"matrix is not square: dimensions are $(size(a))"))
push!(sizes, size(a,1))
end
return sizesSpecifically, reverting just this one makes tests pass Also leaving it lazy and adding a |
|
I'll just revert feceefe for now but we should probably open an issue to track this since it might indicate some deeper issue. |
|
@nanosoldier |
|
@nanosoldier |
|
Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. |
|
The package evaluation job you requested has completed - possible new issues were detected. |
|
Seeing some weird stuff in the PkgEval logs for a few packages: |
This reverts commit 7536f35.
|
@nanosoldier |
|
The package evaluation job you requested has completed - possible new issues were detected. |
|
Are there things remaining for this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backported PRs:
LazyStringinLinearAlgebra.checksquareerror message #53961Need manual backport:
Contains multiple commits, manual intervention needed:
UnionAllunaliasing bug caused by innervars. #53553Non-merged PRs with backport label:
Base.literal_powforAbstractQ#54010