Make Distributed.jl `Worker` struct thread-safe. #38405

vchuravy · 2020-11-12T01:51:50Z

@jpsamaroo and I want to use Distributed in the presence of threading for Dagger. So this is the follow-up to #38134
@JeffBezanson you initially suggested that the Condition is replaced by an Event, but the code seems to want to support multiple state transitions. wait_on_conn re-uses c_state for it's time-out and it not clear to me that timing out there implies timing out other waiters.

Lastly we looked at the gc messages for remote references and made a first attempt at making them safe under multi-threading. Since we can't take locks in a finalizer this implied spawning a task.

cc: @jonas-schulze

stdlib/Distributed/src/cluster.jl

stdlib/Distributed/src/remotecall.jl

vtjnash · 2020-11-20T22:30:05Z

I think #38487 may make this feasible now :)

vchuravy · 2020-11-21T02:16:39Z

I hope so as well.

jonas-schulze · 2020-12-09T23:56:44Z

@vchuravy bump 🙂

Could you please rebase this to give CI another go?

stdlib/Distributed/src/cluster.jl

stdlib/Distributed/src/remotecall.jl

stdlib/Distributed/test/threads.jl

vchuravy · 2021-03-31T18:18:41Z

@JeffBezanson if you have the time I would appreciate a review.

vchuravy · 2021-04-06T03:38:34Z

Seen failure:

      From worker 10:   UNHANDLED TASK ERROR: On worker 1:
      From worker 10:   On worker 14:
      From worker 10:   peer 10 has not connected to 14
      From worker 10:   Stacktrace:
      From worker 10:    [1] error
      From worker 10:      @ ./error.jl:33
      From worker 10:    [2] wait_for_conn
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:194
      From worker 10:    [3] #25
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:182
      From worker 10:    [4] exec_conn_func
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:183
      From worker 10:    [5] exec_conn_func
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:177
      From worker 10:    [6] #118
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278
      From worker 10:    [7] run_work_thunk
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:63
      From worker 10:    [8] macro expansion
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278 [inlined]
      From worker 10:    [9] #117
      From worker 10:      @ ./task.jl:406
      From worker 10:   Stacktrace:
      From worker 10:    [1] #remotecall_fetch#167
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:416 [inlined]
      From worker 10:    [2] remotecall_fetch
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:408
      From worker 10:    [3] #remotecall_fetch#170
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443
      From worker 10:    [4] remotecall_fetch
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443 [inlined]
      From worker 10:    [5] #21
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:169
      From worker 10:    [6] #118
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278
      From worker 10:    [7] run_work_thunk
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:63
      From worker 10:    [8] macro expansion
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278 [inlined]
      From worker 10:    [9] #117
      From worker 10:      @ ./task.jl:406
      From worker 10:   Stacktrace:
      From worker 10:    [1] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 10:      @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:416
      From worker 10:    [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64})
      From worker 10:      @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:408
      From worker 10:    [3] remotecall_fetch(::Function, ::Int64, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 10:      @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443
      From worker 10:    [4] remotecall_fetch
      From worker 10:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443 [inlined]
      From worker 10:    [5] (::Distributed.var"#20#23"{Distributed.Worker})()
      From worker 10:      @ Distributed ./threadingconstructs.jl:170
      From worker 13:   UNHANDLED TASK ERROR: On worker 1:
      From worker 13:   On worker 14:
      From worker 13:   peer 13 has not connected to 14
      From worker 13:   Stacktrace:
      From worker 13:    [1] error
      From worker 13:      @ ./error.jl:33
      From worker 13:    [2] wait_for_conn
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:194
      From worker 13:    [3] #25
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:182
      From worker 13:    [4] exec_conn_func
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:183
      From worker 13:    [5] exec_conn_func
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:177
      From worker 13:    [6] #118
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278
      From worker 13:    [7] run_work_thunk
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:63
      From worker 13:    [8] macro expansion
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278 [inlined]
      From worker 13:    [9] #117
      From worker 13:      @ ./task.jl:406
      From worker 13:   Stacktrace:
      From worker 13:    [1] #remotecall_fetch#167
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:416 [inlined]
      From worker 13:    [2] remotecall_fetch
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:408
      From worker 13:    [3] #remotecall_fetch#170
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443
      From worker 13:    [4] remotecall_fetch
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443 [inlined]
      From worker 13:    [5] #21
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:169
      From worker 13:    [6] #118
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278
      From worker 13:    [7] run_work_thunk
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:63
      From worker 13:    [8] macro expansion
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278 [inlined]
      From worker 13:    [9] #117
      From worker 13:      @ ./task.jl:406
      From worker 13:   Stacktrace:
      From worker 13:    [1] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 13:      @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:416
      From worker 13:    [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64})
      From worker 13:      @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:408
      From worker 13:    [3] remotecall_fetch(::Function, ::Int64, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 13:      @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443
      From worker 13:    [4] remotecall_fetch
      From worker 13:      @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443 [inlined]
      From worker 13:    [5] (::Distributed.var"#20#23"{Distributed.Worker})()
      From worker 13:      @ Distributed ./threadingconstructs.jl:170
      From worker 7:    UNHANDLED TASK ERROR: On worker 1:
      From worker 7:    On worker 14:
      From worker 7:    peer 7 has not connected to 14
      From worker 7:    Stacktrace:
      From worker 7:     [1] error
      From worker 7:       @ ./error.jl:33
      From worker 7:     [2] wait_for_conn
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:194
      From worker 7:     [3] #25
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:182
      From worker 7:     [4] exec_conn_func
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:183
      From worker 7:     [5] exec_conn_func
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:177
      From worker 7:     [6] #118
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278
      From worker 7:     [7] run_work_thunk
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:63
      From worker 7:     [8] macro expansion
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278 [inlined]
      From worker 7:     [9] #117
      From worker 7:       @ ./task.jl:406
      From worker 7:    Stacktrace:
      From worker 7:     [1] #remotecall_fetch#167
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:416 [inlined]
      From worker 7:     [2] remotecall_fetch
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:408
      From worker 7:     [3] #remotecall_fetch#170
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443
      From worker 7:     [4] remotecall_fetch
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443 [inlined]
      From worker 7:     [5] #21
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/cluster.jl:169
      From worker 7:     [6] #118
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278
      From worker 7:     [7] run_work_thunk
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:63
      From worker 7:     [8] macro expansion
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/process_messages.jl:278 [inlined]
      From worker 7:     [9] #117
      From worker 7:       @ ./task.jl:406
      From worker 7:    Stacktrace:
      From worker 7:     [1] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 7:       @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:416
      From worker 7:     [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64})
      From worker 7:       @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:408
      From worker 7:     [3] remotecall_fetch(::Function, ::Int64, ::Int64, ::Vararg{Int64}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      From worker 7:       @ Distributed ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443
      From worker 7:     [4] remotecall_fetch
      From worker 7:       @ ~/builds/julia/usr/share/julia/stdlib/v1.7/Distributed/src/remotecall.jl:443 [inlined]
      From worker 7:     [5] (::Distributed.var"#20#23"{Distributed.Worker})()
      From worker 7:       @ Distributed ./threadingconstructs.jl:170
      From worker 2:    UNHANDLED TASK ERROR: On worker 1:

Sacha0 · 2021-06-22T14:06:55Z

I hope you don't mind Valentin: I took the liberty of rebasing this branch over master. (On 1.7 we consistently hit issues that this branch should address, so we would love to see it over the line. Let me know if there is anything I can do to lend a hand :).)

Sacha0 · 2021-06-22T16:34:08Z

Interestingly this branch does not fully resolve the issues we hit, but it does cause our test suite to fail much earlier/harder and with a better stacktrace 🎉 (as opposed to, without this branch, throwing a more opaque trace in the background and then stalling till timeout).

vchuravy · 2021-06-22T16:47:50Z

Happy to hand this off :) let me know if you want to chat about what needs doing (Happy to have a co-working session on this). I suspect the least will be making the ClusterSerializer threadsafe as well.

vchuravy · 2021-06-22T16:57:14Z

stdlib/Distributed/src/remotecall.jl

-        notify(any_gc_flag)
+        msg = (remoteref_id(rr), myid())
+        # We cannot acquire locks from finalizers
+        @async begin


@Sacha0 I was hoping we would get #39529 for this.

vchuravy · 2021-06-22T17:01:57Z

The MacOS tester hit:

      From worker 3:	AssertionError("wpid > 0")CapturedException(

That's an assertion in process_messages.jl so likely related.

stdlib/Distributed/src/cluster.jl

Makes the worker struct threadsafe as well as flushing the GC messages

vchuravy · 2021-07-16T14:07:37Z

@JeffBezanson I removed unnecessary changes like actually using @spawn everywhere and locked most lookups for w.state

vchuravy · 2021-07-19T12:03:53Z

I am gonna land this, but we should revert if we see any increased flakiness in CI.

JeffBezanson · 2021-07-19T15:27:38Z

stdlib/Distributed/src/cluster.jl


        for wl in wlist
-            (wl.state === W_CREATED) && wait(wl.c_state)
+            if wl.state === W_CREATED


Ah merde...

(cherry picked from commit 5af1cf0)

Revert "fixup to pull request #38405 (#41641)" Seems to be causing hanging in CI testing? This reverts commit 5af1cf0. This reverts commit 5a16805, reversing changes made to 02807b2.

…41722) Also reverts "fixup to pull request #38405 (#41641)" Seems to be causing hanging in CI testing. This reverts commit 5af1cf0 and this reverts commit 5a16805, reversing changes made to 02807b2.

…41722) Also reverts "fixup to pull request #38405 (#41641)" Seems to be causing hanging in CI testing. This reverts commit 5af1cf0 and this reverts commit 5a16805, reversing changes made to 02807b2. (cherry picked from commit 66f9b55)

(cherry picked from commit 740a33a)

…stributed_ts" (JuliaLang/julia#41722) Also reverts "fixup to pull request JuliaLang/julia#38405 (JuliaLang/julia#41641)" Seems to be causing hanging in CI testing. This reverts commit 740a33a and this reverts commit 5a1680533b461471f537f5f5d4ee3c2866b21e1f, reversing changes made to 02807b279a5e6d5acaeb7095e4c0527e2a5c190e. (cherry picked from commit bbc9429)

…41722) Also reverts "fixup to pull request #38405 (#41641)" Seems to be causing hanging in CI testing. This reverts commit 740a33a and this reverts commit 5a16805, reversing changes made to 02807b2.

vchuravy requested review from JeffBezanson and vtjnash November 12, 2020 01:51

jonas-schulze reviewed Nov 13, 2020

View reviewed changes

stdlib/Distributed/src/cluster.jl Show resolved Hide resolved

stdlib/Distributed/src/remotecall.jl Outdated Show resolved Hide resolved

stdlib/Distributed/src/remotecall.jl Outdated Show resolved Hide resolved

This was referenced Nov 13, 2020

Make any_gc_flag threadsafe #38200

Closed

Concurrency violation on interplay between Distributed and Base.Threads JuliaLang/Distributed.jl#73

Closed

vchuravy force-pushed the vc/distributed_ts branch from 3b00732 to 95e6be5 Compare November 16, 2020 02:49

vchuravy force-pushed the vc/distributed_ts branch from 95e6be5 to 810479f Compare December 4, 2020 18:59

jonas-schulze added the backport 1.6 Change should be backported to release-1.6 label Dec 9, 2020

vchuravy force-pushed the vc/distributed_ts branch from 810479f to 20fd68a Compare December 10, 2020 22:01

KristofferC mentioned this pull request Dec 14, 2020

Backports for 1.6-beta #38795

Merged

53 tasks

ViralBShah added the parallelism Parallel or distributed computation label Dec 15, 2020

KristofferC mentioned this pull request Dec 19, 2020

WIP: Backports 1.6-beta #38949

Merged

26 tasks

KristofferC mentioned this pull request Jan 19, 2021

Backports 1.6-rc1 #39160

Merged

60 tasks

KristofferC mentioned this pull request Feb 1, 2021

Backports for 1.6-RC1 #39480

Merged

10 tasks

KristofferC mentioned this pull request Feb 11, 2021

Backports for 1.6-RC2 #39614

Merged

52 tasks

KristofferC mentioned this pull request Mar 14, 2021

Backports release 1.6-RC3 #40029

Merged

14 tasks

KristofferC mentioned this pull request Mar 23, 2021

Backports release 1.6.0 #40150

Merged

10 tasks

vchuravy removed the backport 1.6 Change should be backported to release-1.6 label Mar 23, 2021

vchuravy force-pushed the vc/distributed_ts branch from 20fd68a to 300ed29 Compare March 24, 2021 16:33

jpsamaroo reviewed Mar 24, 2021

View reviewed changes

Sacha0 force-pushed the vc/distributed_ts branch from e287246 to 20a514f Compare June 22, 2021 14:04

Sacha0 added the backport 1.7 label Jun 22, 2021

vchuravy commented Jun 22, 2021

View reviewed changes

vchuravy mentioned this pull request Jun 25, 2021

Use Threads.Condition for package locks #41366

Closed

Sacha0 added the bugfix This change fixes an existing bug label Jun 26, 2021

JeffBezanson force-pushed the vc/distributed_ts branch from 5a64315 to 583004b Compare July 15, 2021 20:31

JeffBezanson added the multithreading Base.Threads and related functionality label Jul 15, 2021

JeffBezanson reviewed Jul 15, 2021

View reviewed changes

stdlib/Distributed/src/cluster.jl Outdated Show resolved Hide resolved

[Distributed] Allow workers to be started with threading enabled

0c073cc

Makes the worker struct threadsafe as well as flushing the GC messages

vchuravy force-pushed the vc/distributed_ts branch from 583004b to 0c073cc Compare July 16, 2021 14:02

vchuravy requested a review from JeffBezanson July 16, 2021 14:07

vchuravy mentioned this pull request Jul 17, 2021

Concurrency violation detected on multithread config JuliaParallel/Dagger.jl#229

Closed

vchuravy merged commit 5a16805 into master Jul 19, 2021

vchuravy deleted the vc/distributed_ts branch July 19, 2021 12:04

JeffBezanson reviewed Jul 19, 2021

View reviewed changes

vchuravy added a commit that referenced this pull request Jul 19, 2021

fixup! Merge pull request #38405 from JuliaLang/vc/distributed_ts

1191cd0

JeffBezanson pushed a commit that referenced this pull request Jul 19, 2021

fixup to pull request #38405 (#41641)

5af1cf0

KristofferC pushed a commit that referenced this pull request Jul 26, 2021

fixup to pull request #38405 (#41641)

f3da8d9

(cherry picked from commit 5af1cf0)

vtjnash mentioned this pull request Jul 28, 2021

Revert "Merge pull request #38405 from JuliaLang/vc/distributed_ts" #41722

Merged

KristofferC removed the backport 1.7 label Aug 3, 2021

jpsamaroo mentioned this pull request Sep 5, 2021

[Distributed] error in running finalizer: ConcurrencyViolationError(msg="lock must be held") (any_gc_flag) #42126

Closed

vchuravy added a commit to JuliaLang/Distributed.jl that referenced this pull request Oct 6, 2023

fixup to pull request JuliaLang/julia#38405 (JuliaLang/julia#41641)

b14100c

(cherry picked from commit 740a33a)

Keno pushed a commit that referenced this pull request Jun 5, 2024

fixup to pull request #38405 (#41641)

740a33a

Uh oh!

Make Distributed.jl Worker struct thread-safe. #38405

Make Distributed.jl Worker struct thread-safe. #38405

Uh oh!

Conversation

vchuravy commented Nov 12, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vtjnash commented Nov 20, 2020

Uh oh!

vchuravy commented Nov 21, 2020

Uh oh!

jonas-schulze commented Dec 9, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vchuravy commented Mar 31, 2021

Uh oh!

vchuravy commented Apr 6, 2021

Uh oh!

Sacha0 commented Jun 22, 2021

Uh oh!

Sacha0 commented Jun 22, 2021

Uh oh!

vchuravy commented Jun 22, 2021

Uh oh!

vchuravy Jun 22, 2021

Choose a reason for hiding this comment

Uh oh!

vchuravy commented Jun 22, 2021

Uh oh!

Uh oh!

vchuravy commented Jul 16, 2021

Uh oh!

vchuravy commented Jul 19, 2021

Uh oh!

JeffBezanson Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

vchuravy Jul 19, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Make Distributed.jl `Worker` struct thread-safe. #38405

Make Distributed.jl `Worker` struct thread-safe. #38405