-
-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Non-reproducibility of @threads when varying number of threads #49064
Comments
There is nothing entirely notable here, since the order of computation (with nthreads>1) is not defined, so the value of the last computation is not defined as part of the But this also seems to be a curious case, since the number of Tasks spawned is dependent on a configuration parameter of |
This is slightly beside the point, because if I manage my own collection of RNGs -- and in fact this is how we solve the reproducibility issue in our application -- I can make the MWE work using Random
import Base.Threads.@threads
println("Number of threads: $(Threads.nthreads())")
const n_iters = 10000;
result = zeros(n_iters);
rngs = [MersenneTwister(i) for i in 1:n_iters]
@threads for i in 1:n_iters
# in a real problem, do some expensive calculation here...
result[i] = rand(rngs[i]);
end
println("Result: $(last(result))") Now the result is invariant to
|
|
Using using Random
using SplittableRandoms: SplittableRandom, split
import Base.Threads.@threads
println("Number of threads: $(Threads.nthreads())")
const n_iters = 10000;
const master_rng = SplittableRandom(1)
result = zeros(n_iters);
rngs = [split(master_rng) for _ in 1:n_iters]
@threads for i in 1:n_iters
# in a real problem, do some expensive calculation here...
result[i] = rand(rngs[i]);
end
println("Result: $(last(result))")
|
I'm working on a PR that would leave the parent task's RNG unchanged by the number of child tasks it spawns. This requires keeping a "child counter" in the parent task and seeding the child task's RNG with a combination of the parent task's RNG state and the counter. Splitting off a child only advances the parent's counter, it does not affect the parent's RNG state. |
That's interesting! However, I don't think that particular fix in and of itself will make I want to re-iterate that we're not in any way advocating for |
parent = Xoshiro(); # note that period is 2^256
xos = Vector{Xoshiro}(undef, 8);
for i = 1:8
xos[i] = copy(parent)
long_jump!(parent) # advance by 2^128
end Assuming that each thread uses a single RNG instance, it's virtually impossible for their states to overlap. In general, if one needs to control RNG state and use threads, the only option is to deterministically equip each task (thread) with its own RNG in the manner that one desires. As an aside, it's probably worth taking a look at Vigna's papers so that one can make an informed decision. |
Having jump polynomials for Xoshiro instead of the current ad-hoc approach to However, that would not by itself fix the varying-nthreads-non-reproducibility of @threads---which, again, we don't argue has to be fixed, it should only be clarified. Indeed, by replacing using Random, Base.Threads
println("Number of threads: $(Threads.nthreads())")
const n_iters = 10000;
result = zeros(n_iters);
Random.seed!(1);
@elapsed @sync for i in 1:n_iters
# in a real problem, do some expensive calculation here...
@spawn result[i] = rand();
end
println("Result: $(last(result))")
The reason that this works is simply that the number of tasks == number of PRNG stream == and thus it is truly independent of |
Hi -- first of all, thank you for an amazing piece of software!
The documentation for
TaskLocalRNG
readsHowever, when using the
@threads
macro in a way that does not explicitly make use ofnthreads()
, we (cc: @alexandrebouchard) still get different results when changing the number of threads (see MWE at the bottom).We realized that the reason is that
@threads
createsnthreads()
tasks -- and thereforenthreads()
RNG streams -- so it is no surprise that the results change with the number of threads.Since
@threads
is perhaps the most widely used approach to multithreading in Julia, we think that there should be a!!! warning
in the documentation for@threads
. It should state that thenthreads()
--invariance guarantee fromTaskLocalRNG
is voided because of the reason outlined above.Looking forward to reading your opinion on this issue.
Best,
Miguel
versioninfo()
For 8 threads this gives
whereas for 1 thread the result is
The text was updated successfully, but these errors were encountered: