You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am unsure how easy this is to do, and it may be that it requires manual checks for every learner, but it would be useful to have some sort of error handling or checks to prevent conflicting parallelisation methods. This is related to #501 and possible #542.
For example, I ran a large benchmark experiment that kept crashing without me realising, the first time because a learner was being used when OpenMP was automatically enabled by its package, the second time because the learner was interfaced via reticulate, which future specifically says cannot be parallelised.
Some ideas for how we could deal with this, in increasingly complexity:
Add a line in the @details section of docs - but people don't read this and then may not realise why the experiment never ends
Write manual checks for each learner to detect which future::plan is used - e.g. in train
Same as 2. but if conflict detected then call future::plan("sequential")
The text was updated successfully, but these errors were encountered:
Some ideas for how we could deal with this, in increasingly complexity:
1. Add a line in the `@details` section of docs - but people don't read this and then may not realise why the experiment never ends
2. Write manual checks for each learner to detect which `future::plan` is used - e.g. in `train`
3. Same as 2. but if conflict detected then call `future::plan("sequential")`
Some remote backends work nicely with nested parallelization (e.g., batchtools + basically everything else), others do not. I can implement (2) and (3) only if we come up with a list of combinations which are known to not work together, or is it only multicore + OpenMP?
I am unsure how easy this is to do, and it may be that it requires manual checks for every learner, but it would be useful to have some sort of error handling or checks to prevent conflicting parallelisation methods. This is related to #501 and possible #542.
For example, I ran a large benchmark experiment that kept crashing without me realising, the first time because a learner was being used when OpenMP was automatically enabled by its package, the second time because the learner was interfaced via
reticulate
, whichfuture
specifically says cannot be parallelised.Some ideas for how we could deal with this, in increasingly complexity:
@details
section of docs - but people don't read this and then may not realise why the experiment never endsfuture::plan
is used - e.g. intrain
future::plan("sequential")
The text was updated successfully, but these errors were encountered: