-
Notifications
You must be signed in to change notification settings - Fork 0
Replace furrr with mirai for parallel execution #77
Copy link
Copy link
Closed
Description
Replace furrr with mirai for parallel execution
Problem
furrr/future creates heavyweight worker processes per future_map() call. Workers are ephemeral — no connection reuse, serialization overhead. Each worker must call frs_db_conn() independently and relies on PG_*_SHARE env vars.
Proposed Solution
Switch to mirai for parallel map in frs_habitat(). mirai provides:
- Lightweight persistent daemons via NNG (nanomsg)
mirai_map()for parallel map- Lower overhead than future/furrr
- Foundation for crew.aws.batch (crew.aws.batch for province-wide parallel pipeline #76) — crew is built on mirai
Scope
- Replace ~20 lines of furrr/future code in
R/frs_habitat.R - Update DESCRIPTION: remove furrr/future from Suggests, add mirai
- Pass DB connection params to workers (not env-var dependent)
- Two parallel sections: Phase 1 (partition prep) and Phase 2 (species classification)
Relates to
- crew.aws.batch for province-wide parallel pipeline #76 — crew.aws.batch (mirai is prerequisite)
- Profile and optimize Phase 2 species classification performance #72 — performance optimization
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels