Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backports for 1.9.0-rc2 #48935

Merged
merged 46 commits into from
Mar 31, 2023
Merged

Backports for 1.9.0-rc2 #48935

merged 46 commits into from
Mar 31, 2023

Conversation

KristofferC
Copy link
Sponsor Member

@KristofferC KristofferC commented Mar 7, 2023

Backported PRs:

Need manual backport:

Contains multiple commits, manual intervention needed:

Non-merged PRs with backport label:

Keno and others added 4 commits March 7, 2023 16:19
We were accidentally dropping the effects of calls from
`iterate` calls performed during abstract_iteration. This
allowed calls that were not actually eligible for (semi-)concrete
evaluation to go through that path anyway. This could cause
incorrect results (see test), though it was usually fine, since
iterate call tend to not have side effects. It was noticed
however in #47688, because it forced irinterp down a path
that was not meant to be reachable (resulting in a TODO
error message). For good measure, let's also address this
todo (since it is reachable by external absint if they want),
but the missing effect propagation was the more serious bug
here.

(cherry picked from commit 2a0d58a)
Rather than a binary valid/not-valid, we track the exact later world
that deleted it, relative to when we first assumed it may be valid.

(cherry picked from commit 4c32381)
@KristofferC KristofferC added the kind:release Release management and versioning. label Mar 7, 2023
KristofferC and others added 14 commits March 10, 2023 15:35
skip checked `equal_union` by set `state[i] = 1`
…48441)

* Avoid earsing `Runion` within nested `forall_exists_subtype`

If `Runion.more != 0` we‘d better not erase the local `Runion` as we need it if the subtyping fails after.

This commit replaces `forall_exists_subtype` with a local version.
It first tries `forall_exists_subtype` and estimates the "problem scale".
If subtyping fails and the scale looks small then it switches to the slow path.

TODO: At present, the "problem scale" only counts the number of checked `Lunion`s.
But perhaps we need a more accurate result (e.g. sum of `Runion.depth`)

* Change the reversed subtyping into a local check.

Make sure we don't forget the bound in `env`.
(And we can fuse `local_forall_exists_subtype`)

* Optimization for non-union invariant parameter.
If `y` has no free typevar. Then `x <:  y` wont cause any env change. Thus there's no need to split `y`.
Allow specifying which thread pool's size to retrieve.

(cherry picked from commit dc68ffc)
Now returns `threadpoolsize(pool)`.

(cherry picked from commit 944dce9)
If a task is spawned with `:interactive` but there are no interactive
threads, set the task's thread pool to `:default` so that we don't have
to keep checking it in other places.

(cherry picked from commit 38727be)
If there's only one thread in the task's preferred thread pool, use that
thread's work queue.

(cherry picked from commit a3a92e8)
(cherry picked from commit 55422d9)
(cherry picked from commit 4ea2d2f)
We use the specialized signature of a method for accurate analysis of
whether a static parameter is constrained or not. However, it turns out
that we can only use it when it doesn't contain any free type variables
(which it sometimes does, e.g., when the inference entry is given a
signature with a free type variable). In such cases, we should use the
method signature, which, by design, never contains any free type
variables.
@timholy
Copy link
Sponsor Member

timholy commented Mar 24, 2023

If #48841 is being backported, we also need #48982.

udohjeremiah and others added 8 commits March 24, 2023 15:01
)

* Install 7z, lld, dsymutil to private libexec/julia directory

Installing files directly into /usr/libexec pollutes the install tree
and can create conflicts if other apps do the same. Instead, install
them to `private_libdir`, which defaults to `$(libexecdir)/julia`.
This is similar to what we do with `private_libdir`.

(cherry picked from commit eb2e968)
Avoid possible nested `Vararg`.

Fix #48950

(cherry picked from commit 4da0359)
Apparently we never actually implemented support for invalidation
detection on invoke edges, and furthermore, it had erased part of the
support for regular edges.

Generalize our code for detecting replacement of a method, to be used
when computing replacement of an invoke edge.

Fix #48802

(cherry picked from commit 386b1b6)
Since it does not compute and branch on typeof, in my measurements, this
implementation is up to 10x faster!

(cherry picked from commit 369660f)
vtjnash and others added 7 commits March 24, 2023 15:01
Quiets an error on CI when running missing tests.

Refs #45823

(cherry picked from commit 3919a89)
To interpret causes of invalidation, it's helpful to understand what
signature got invalidated. #48954 inadvertently dropped this info
from the logging stream; this commit restores it.

(cherry picked from commit 99fce41)
Co-authored-by: Jameson Nash <vtjnash@gmail.com>
(cherry picked from commit 1323d2f)
(cherry picked from commit 6d678fe)
@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(ALL, vs = ":release-1.9", configuration = (buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],), vs_configuration=(buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],))

@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runbenchmarks(ALL, vs = ":release-1.9")

@nanosoldier
Copy link
Collaborator

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here.

@nanosoldier
Copy link
Collaborator

Your package evaluation job has completed - possible new issues were detected.
A full report can be found here.

@KristofferC
Copy link
Sponsor Member Author

@N5N3, it seems the subtype backport fixes caused a bunch of packages to start failing. For example:


     Testing LogParser
     Testing Running tests...
WARNING: method definition for catvaluetype at /home/kc/.julia/packages/CategoricalArrays/dmrjI/src/value.jl:48 declares type variable R but does not use it.
julia: /home/kc/julia1.9/src/subtype.c:1955: ijl_subtype_env: Assertion `obvious_subtype == 3 || obvious_subtype == subtype || jl_has_free_typevars(x) || jl_has_free_typevars(y)' failed.

[539611] signal (6.-6): Aborted
in expression starting at /home/kc/.julia/packages/CategoricalArrays/dmrjI/src/value.jl:100
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fd21be2871a)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
ijl_subtype_env at /home/kc/julia1.9/src/subtype.c:1955
jl_type_intersection_env_s at /home/kc/julia1.9/src/subtype.c:3553
jl_type_intersection2 at /home/kc/julia1.9/src/gf.c:1813
ijl_method_table_insert at /home/kc/julia1.9/src/gf.c:1977
ijl_method_def at /home/kc/julia1.9/src/method.c:1101
eval_methoddef at /home/kc/julia1.9/src/interpreter.c:111
eval_body at /home/kc/julia1.9/src/interpreter.c:565
jl_interpret_toplevel_thunk at /home/kc/julia1.9/src/interpreter.c:762
jl_toplevel_eval_flex at /home/kc/julia1.9/src/toplevel.c:912
jl_toplevel_eval_flex at /home/kc/julia1.9/src/toplevel.c:856
ijl_toplevel_eval_in at /home/kc/julia1.9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1866
_jl_invoke at /home/kc/julia1.9/src/gf.c:2739 [inlined]
ijl_apply_generic at /home/kc/julia1.9/src/gf.c:2940
_include at ./loading.jl:1926
include at ./Base.jl:457
jfptr_include_31723 at /home/kc/julia1.9/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /home/kc/julia1.9/src/gf.c:2739 [inlined]
ijl_apply_generic at /home/kc/julia1.9/src/gf.c:2940
jl_apply at /home/kc/julia1.9/src/julia.h:1878 [inlined]
jl_f__call_latest at /home/kc/julia1.9/src/builtins.c:774
include at /home/kc/.julia/packages/CategoricalArrays/dmrjI/src/CategoricalArrays.jl:1
unknown function (ip: 0x7fd208719212)
_jl_invoke at /home/kc/julia1.9/src/gf.c:2739 [inlined]
ijl_apply_generic at /home/kc/julia1.9/src/gf.c:2940
jl_apply at /home/kc/julia1.9/src/julia.h:1878 [inlined]
do_call at /home/kc/julia1.9/src/interpreter.c:126
eval_value at /home/kc/julia1.9/src/interpreter.c:226
eval_stmt_value at /home/kc/julia1.9/src/interpreter.c:177 [inlined]
eval_body at /home/kc/julia1.9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /home/kc/julia1.9/src/interpreter.c:762
jl_toplevel_eval_flex at /home/kc/julia1.9/src/toplevel.c:912
jl_eval_module_expr at /home/kc/julia1.9/src/toplevel.c:203 [inlined]

@N5N3
Copy link
Member

N5N3 commented Mar 29, 2023

I‘ll take a look today.
Edit: MWE:

julia> S = Tuple{Type{Missing}, Missing}
Tuple{Type{Missing}, Missing}

julia> T1 = Tuple{Type{Union{Base.Missing, T}}, T} where T<:Array
Tuple{Type{Union{Missing, T}}, T} where T<:Array

julia> SubtypeToy.dump_subtype(S, T1) # <: but skip `obvious_subtype`
0

julia> T2 = Tuple{Type{Union{Base.Missing, T}}, T} where T<:Union{Array,Int}
Tuple{Type{Union{Missing, T}}, T} where T<:Union{Int64, Array}

julia> SubtypeToy.dump_subtype(S, T2)
1

Looks like simple_meet failled to give Union{Array,Int}Missing == Union{}

vtjnash and others added 11 commits March 30, 2023 08:35
We see Windows CI fail often here, and we don't actually need this
memory--only the first thread should initialize it. (indeed, we were
temporarily corrupting these data-structures in the process by
reinitializing them, though that was not typically detectable.)

    From worker 7:  Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffa26726e87 -- jl_makecontext at C:/workdir/src\win32_ucontext.c:67
    From worker 7:  in expression starting at C:\buildkite-agent\builds\win2k22-amdci6-6\julialang\julia-master\julia-ceffaee345\share\julia\test\ccall.jl:1066
    From worker 7:  jl_makecontext at C:/workdir/src\win32_ucontext.c:67
    From worker 7:  jl_install_thread_signal_handler at C:/workdir/src\signals-win.c:490
    From worker 7:  jl_init_root_task at C:/workdir/src\task.c:1568
    From worker 7:  ijl_adopt_thread at C:/workdir/src\threading.c:414
    From worker 7:  unknown function (ip: 000001d791a58969)
    From worker 7:  uv__queue_work at /workspace/srcdir/libuv\src\threadpool.c:305
    From worker 7:  worker at /workspace/srcdir/libuv\src\threadpool.c:122
    From worker 7:  uv__thread_start at /workspace/srcdir/libuv\src/win\thread.c:111
    From worker 7:  beginthreadex at C:\Windows\System32\msvcrt.dll (unknown line)
    From worker 7:  endthreadex at C:\Windows\System32\msvcrt.dll (unknown line)
    From worker 7:  BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line)
    From worker 7:  RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
    From worker 7:  Allocations: 352796158 (Pool: 352389694; Big: 406464); GC: 722

(cherry picked from commit 685da8f)
The dynamic linker needs to unify `llvm::Any::TypeId` across DSOs. In our case
`libjulia-codegen` and `libLLVM`.

See https://github.com/llvm/llvm-project/blob/2bc4c3e920ee078ef2879b00c40440e0867f0b9e/llvm/include/llvm/ADT/Any.h#L30

Fixes: #49121
(cherry picked from commit d8fa3c8)
Let Base handle concatenation of arrays and numbers

(cherry picked from commit e6c84a1)
* Add ITTAPI sources for offline build

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
(cherry picked from commit 6b934f9)
Co-authored-by: Jameson Nash <vtjnash@gmail.com>
(cherry picked from commit bc33c81)
(cherry picked from commit 40692cca1619a06991bebabc162e531255d99ddd)
(cherry picked from commit d586b0c)
(cherry picked from commit 450c1fa)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:release Release management and versioning.
Projects
None yet
Development

Successfully merging this pull request may close these issues.