mirai

ミライ

みらい 未来

Minimalist Async Evaluation Framework for R

→ Run R code in parallel without blocking your session

→ Distribute workloads across local or remote machines

→ Perform actions reactively as soon as tasks complete

Installation

install.packages("mirai")

Quick Start

mirai() evaluates an R expression asynchronously in a parallel process.

daemons() sets up persistent background processes for parallel computations.

library(mirai)
daemons(5)

m <- mirai({
  Sys.sleep(1)
  100 + 42
})

mp <- mirai_map(1:9, \(x) {
  Sys.sleep(1)
  x^2
})

m
#> < mirai [] >
m[]
#> [1] 142

mp
#> < mirai map [4/9] >
mp[.flat]
#> [1]  1  4  9 16 25 36 49 64 81

daemons(0)

Design Philosophy

→ Modern Foundation

• Architected on current communication technologies (IPC, TCP, secure TLS)
• Professional queueing and scheduling built on nanonext and NNG
• Engineered for custom serialization of cross-language data formats (e.g. torch, Arrow)

→ Extreme Performance

• Scales to millions of tasks across thousands of processes
• 1,000x greater efficiency and responsiveness than previously possible
• Zero-latency, event-driven promises brings modern async to R

→ Production First

• 100% reliable abstraction of the multi-process, message-passing paradigm
• Consistent and transparent evaluation, from minimal complexity and no hidden state
• Observability at scale through OpenTelemetry integration

→ Deploy Everywhere

• Deploy across local, remote (SSH), and HPC environments (Slurm, SGE, PBS, LSF)
• Combine and extend local processing power with network or cloud compute
• Direct tasks to the most suitable resources, using modular compute profiles

Powers the R Ecosystem

mirai serves as a foundation for asynchronous, parallel and distributed computing in the R ecosystem.

  The first official alternative communications backend for R, the ‘MIRAI’ parallel cluster, a feature request by R-Core.

  Powers parallel map for purrr, a core tidyverse package.

  Primary async backend for Shiny with full ExtendedTask support.

  Built-in async evaluator enabling the @async tag in plumber2.

  Core parallel processing infrastructure provider for tidymodels.

  Seamless use of torch tensors, models and optimizers across parallel processes.

  Query databases over ADBC connections natively in the Arrow data format.

  R Polars leverages mirai’s serialization registration mechanism for transparent use of Polars objects.

  Targets uses crew as its default high-performance computing backend. Crew is a distributed worker launcher extending mirai to different computing platforms.

Acknowledgements

Will Landau for being instrumental in shaping development of the package, from initiating the original request for persistent daemons, through to orchestrating robustness testing for the high performance computing requirements of crew and targets.

Joe Cheng for integrating the ‘promises’ method to work seamlessly within Shiny, and prototyping event-driven promises.

Luke Tierney of R Core, for discussion on L’Ecuyer-CMRG streams to ensure statistical independence in parallel processing, and reviewing mirai’s implementation as the first ‘alternative communications backend for R’.

Travers Ching for a novel idea in extending the original custom serialization support in the package.

Hadley Wickham, Henrik Bengtsson, Daniel Falbel, and Kirill Müller for many deep insights and discussions.

Links

mirai | nanonext | CRAN HPC Task View

–

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.