Skip to content

updated helpers for parallel processing - #1028

Merged
topepo merged 12 commits into
melodie-switchfrom
mirai
Jul 2, 2025
Merged

topepo merged 12 commits into
melodie-switchfrom
mirai

Conversation

@topepo

@topepo topepo commented Jun 30, 2025

Copy link
Copy Markdown
Member

Some basic helper functions to determine when and how to use parallel processing.

Docs are updated too.

The next step is to update loop_call() with the additional logic. I'll make updates to this branch for that.

Comment thread R/parallel_new.R

# ------------------------------------------------------------------------------

has_non_par_pkgs <- function(object, control, verbose = FALSE) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that we'll expose verbose to users but we might want it internally.

Comment thread R/parallel_new.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
@topepo
topepo marked this pull request as ready for review July 1, 2025 18:27
@topepo

topepo commented Jul 1, 2025

Copy link
Copy Markdown
Member Author

We are in the middle of major surgery with this repo so I would concentrate on these two files:

@topepo
topepo requested a review from simonpcouch July 1, 2025 18:29

@simonpcouch simonpcouch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable!

Leaving review as Comment not to request re-review before merging or anything, but just to indicate that I don't have enough familiarity with the codebase at this point to confidently endorse!

Comment thread NEWS.md Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
Comment thread R/parallel_new.R
}
mirai_avail <- function() {
any(search() == "package:mirai")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible to set daemons without loading the mirai package, and same to plan without future loaded. Could the logic here just be based on the information in future::nbrOfWorkers() and mirai::status()? Checking whether the packages are loaded feels like an unusual mechanism.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is for both mirai and future to be in suggests so someone will have to install both even if they don't want to run in parallel.

@shikokuchuo shikokuchuo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, I've provided some suggestions in the comments.

Comment thread tests/testthat/test-parallel.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread R/parallel_new.R Resolved
Comment thread tests/testthat/test-parallel.R Resolved
topepo and others added 8 commits July 2, 2025 07:52
Co-authored-by: Simon P. Couch <simonpatrickcouch@gmail.com>
Co-authored-by: Charlie Gao <53399081+shikokuchuo@users.noreply.github.com>
Co-authored-by: Charlie Gao <53399081+shikokuchuo@users.noreply.github.com>
Co-authored-by: Simon P. Couch <simonpatrickcouch@gmail.com>
Co-authored-by: Simon P. Couch <simonpatrickcouch@gmail.com>
Co-authored-by: Simon P. Couch <simonpatrickcouch@gmail.com>
@topepo
topepo merged commit 15a42e6 into melodie-switch Jul 2, 2025
@topepo
topepo deleted the mirai branch July 2, 2025 13:00
@topepo topepo mentioned this pull request Jul 7, 2025
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

@github-actions github-actions Bot locked and limited conversation to collaborators Jul 17, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants