Skip to content

Fix list-valued generate/analyse on parallel workers (future + socket)#91

Merged
philchalmers merged 3 commits into
philchalmers:mainfrom
emstruong:fix/list-generate-analyse-parallel
Jun 24, 2026
Merged

Fix list-valued generate/analyse on parallel workers (future + socket)#91
philchalmers merged 3 commits into
philchalmers:mainfrom
emstruong:fix/list-generate-analyse-parallel

Conversation

@emstruong

@emstruong emstruong commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Fix list-valued generate/analyse on parallel workers (future + socket)

Claude wrote this, I don't have the understanding of how parallel processing in R works to review this, sorry. But I thought perhaps it might save you time by having Claude's proposed changes + tests.

Closes #90.

What

Make list-valued generate/analyse work on every back-end. Two gaps caused fatal
terminations when the function lists (stashed in SimDesign:::.SIMDENV) were not
delivered to parallel workers:

  1. parallel = "future" — neither GENERATE_FUNCTIONS nor ANALYSE_FUNCTIONS
    reached the workers (future's automatic global detection does not export
    package-internal objects), so the run aborted with
    object 'ANALYSE_FUNCTIONS' not found (or 'GENERATE_FUNCTIONS').
  2. parallel = TRUE (socket/MPI) — a list-valued generate aborted with
    object 'GENERATE_FUNCTIONS' not found, because clusterExport() shipped
    ANALYSE_FUNCTIONS but never GENERATE_FUNCTIONS.

Serial (parallel = FALSE) was always fine.

Why

When generate/analyse are lists, runSimulation() substitutes the internal
combined_Generate()/combined_Analyses() dispatchers and stores the original
function lists in the package-internal .SIMDENV. Workers load a fresh copy of the
package whose .SIMDENV is empty, so the lists must be sent to them explicitly. One
list was missing from the socket export, and the future path did not export them at
all.

How

  • R/runSimulation.R: add a local GENERATE_FUNCTIONS (mirroring ANALYSE_FUNCTIONS)
    and a guarded clusterExport(cl, "GENERATE_FUNCTIONS").
  • R/analysis.R: in the future branch of Analysis(), wrap the worker function so
    every future restores the captured function lists into the worker's .SIMDENV via
    getFromNamespace() before the replication runs — correct for any future plan.

Tests

tests/tests/test-01-core.R now exercises list-valued generate and list-valued
analyse across serial, socket (parallel = TRUE) and future (multisession)
back-ends, asserting no FATAL_TERMINATION.

Verification

Built and installed the patched package; both reprexes return FATAL_TERMINATION = NULL and correct output columns across 3/3 trials on each affected back-end. The same
reprexes fail deterministically (3/3) on the released package.

Note for reviewers: reproduce with the installed package, not devtools::load_all()
load_all() masks the future failure because the dev namespace makes future
serialise .SIMDENV as an ordinary global.

emstruong and others added 2 commits June 22, 2026 08:24
…ckends

A list-valued `generate` aborted under `parallel = TRUE` with
"object 'GENERATE_FUNCTIONS' not found": runSimulation() exported
ANALYSE_FUNCTIONS to the cluster workers but never GENERATE_FUNCTIONS, so
combined_Generate() could not find the function list on the workers.

Introduce a local GENERATE_FUNCTIONS (mirroring ANALYSE_FUNCTIONS) and add a
guarded clusterExport() for it.

Also expand the regression test to exercise both list-valued generate and
list-valued analyse across the serial, socket (parallel = TRUE) and future
(multisession) backends, covering this fix and the earlier future-backend fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@philchalmers philchalmers left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's awkward to do the redefinition of mainsim, but I don't see away around this since future doesn't expose the cluster info. Could you add this information to the NEWS.md file?

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@emstruong

Copy link
Copy Markdown
Contributor Author

Hi @philchalmers , I asked Claude to give it a shot.

@philchalmers

Copy link
Copy Markdown
Owner

Good enough for me. I'll make any edits after the merge. Thanks!

@philchalmers philchalmers merged commit 0402753 into philchalmers:main Jun 24, 2026
3 checks passed
@emstruong emstruong deleted the fix/list-generate-analyse-parallel branch June 24, 2026 13:47
philchalmers added a commit that referenced this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

List-valued generate/analyse fail under parallel workers (future backend, and parallel=TRUE for generate)

2 participants