[Feature Request] Parallelization and seeds?

[emstruong]

Hello again,

I was wondering if the parallel execution of execute_multiverse() could be extended to have a seed = TRUE option? Sometimes there may be random-number-generation happening in particular branches and having the option of setting the seed to be either different or the explicitly identical across the multiverse would be nice.

[abhsarma]

Unless I am missing something, can't you do something like this?

M = multiverse()

inside(M, {
  one = branch(assignments, 
    "a"  ~ rnorm(5, 1, 1),
    "b" ~ rnorm(5, 0, 1),
    "c" ~ rnorm(5, -1, 1)
    )
})

set.seed(1234)
execute_multiverse(M)

extract_variables(M, one)$one # each of the three branches will have a different set of random numbers

M2 = multiverse()

inside(M2, {
  one = branch(assignments, 
    "a"  ~ rnorm(5, 1, 1),
    "b" ~ rnorm(5, 0, 1),
    "c" ~ rnorm(5, -1, 1)
    )
})

set.seed(1234)
execute_multiverse(M2)

extract_variables(M2, one)$one # each of the three branches will have the same set of random numbers as `M`

I am not a 100% sure how R environments work with respect to seeds, but basically what is going on here is that as long as the seed is specified right before execute_multiverse is run, it should yield the same set of random numbers. Is that the type of behaviour you are looking for?

If so, then I can add an additional seed argument to execute_multiverse which basically does this so that things are more easily replicable by others.

[emstruong]

I think I'm looking for two possibilities.

The first is that when running the below

M = multiverse()

inside(M, {
  one = branch(assignments, 
    "a"  ~ rnorm(5, 1, 1),
    "b" ~ rnorm(5, 1, 1),
    "c" ~ rnorm(5, 1, 1)
    )
})

execute_multiverse(M, seed = 1234, parallel = TRUE)

extract_variables(M, one)$one

To have the ability for all the one to be identical to each other.

This is helpful to distinguish the variability in results due to what decision was taken; and variability in results due to what seed was obtained. In my opinion, given the nature of multiverse analyses, regardless of parallelization, random number generation should probably be identical between multiverses by default.

---

The second possibility is the opposite, but a careful one.

M = multiverse()

inside(M, {
  one = branch(assignments, 
    "a"  ~ rnorm(5, 1, 1),
    "b" ~ rnorm(5, 1, 1),
    "c" ~ rnorm(5, 1, 1)
    )
})

execute_multiverse(M, seed = 1234, parallel = TRUE, )

extract_variables(M, one)$one

Where all the ones are both different from each other, but also uncorrelated to each other, despite being executed in parallel.

I don't feel competent enough in the intricacies of RNG to comment on how the two above may be achieved and whether they should be achieved.

However, I will note these three articles:

1. https://www.jottr.org/2020/09/22/push-for-statistical-sound-rng/
2. https://future.apply.futureverse.org/reference/future_lapply.html#reproducible-random-number-generation-rng-
3. https://en.wikipedia.org/wiki/RANDU

[abhsarma]

Okay.

1. In the first scenario that you are describing, why declare rnorm or any other random number generator function within the multiverse? I think this would be easily solved by declaring it outside the multiverse because you want it to be consistent.

In my opinion, given the nature of multiverse analyses, regardless of parallelization, random number generation should probably be identical between multiverses by default

I can see the merits of what you are suggesting here, but I actually feel that if a simple RNG is yielding different results, then those results should be discarded.

2. This is the behaviour that should be implemented---I'll look into it.

[emstruong]

1. In the first scenario that you are describing, why declare `rnorm` or any other random number generator function within the multiverse? I think this would be easily solved by declaring it outside the multiverse because you want it to be consistent.

That is because there are certain statistical procedures or algorithms that importantly make use of random number generation to perform their analysis. For example, random forests randomly split the data into bootstrap samples to perform their analysis.

Some more complex algorithms also importantly depend on the randomly generated starting values. I know Generalized Structured Component Analysis is one such example, and I'm fairly confident that at least some versions of neural networks have this feature.

In my opinion, given the nature of multiverse analyses, regardless of parallelization, random number generation should probably be identical between multiverses by default

I can see the merits of what you are suggesting here, but I actually feel that if a simple RNG is yielding different results, then those results should be discarded.

Please see above for why I think RNG is important. If we for example did a multiverse analysis of all the different random forest fitting options, then consistent RNG may be important. Though, regardless of its importance, I think software should be relatively liberal in allowing users what to do or not, prescription can be done via the default values. Ultimately, I think it's an open research question about whether the RNG should or should be not be the same between multiverse analyses.

2. This is the behaviour that should be implemented---I'll look into it.

👍

[abhsarma]

Sure, I am not denying that it might not be useful for each universe in a multiverse to have the same set of numbers generated by a RNG process. My point is that, if that is the desired behaviour, then it should be handled outside the multiverse. Thinking more about it, I see two ways:

1. specify the RNG outside the multiverse
2. specify the seed within each universe of the multiverse, right before where the RNG is called. (That said, I'll need to make sure the optimisation we do in terms of pruning redundant computation does not interfere with this)