Skip to content
Framework for large distributed pipelines
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cpp/tasklib [tests][cpp] Cpp executor updated Jun 7, 2018
dashboard [dashboard] Small visual improvement Aug 28, 2018
docs [docs] Link to tutorial Part 3 Oct 2, 2018
python Fix test fixture, fix new lints, add readme mimetype Dec 16, 2018
rain_core RF [server] Removed #[macro_use] Oct 29, 2018
rain_server RF [server] Removed #[macro_use] Oct 29, 2018
rain_task RF [server] Removed #[macro_use] Oct 29, 2018
rain_task_test RF [server] Removed #[macro_use] Oct 29, 2018
tests/pytests
utils [tooling] Exoscale deployment script updated Sep 10, 2018
.flake8 Fix test fixture, fix new lints, add readme mimetype Dec 16, 2018
.gitignore Prerelease preparation Aug 29, 2018
.gitlab-ci.yml
.travis.yml FIX [CI] Travis CI Jun 14, 2018
CHANGELOG.md Prerelease preparation Aug 29, 2018
Cargo.lock [server] Migrated from tokio-io::codec to tokio::codec Oct 29, 2018
Cargo.toml Releasing version 0.4.0 Sep 5, 2018
Dockerfile FIX [CI] Travis CI Jun 14, 2018
LICENSE License added Feb 26, 2018
README.md Readme fixed Dec 16, 2018
requirements.readthedocs.txt FIX readthedocs requirements Jul 2, 2018

README.md

Rain

PyPI Crates.io Build Status Gitter

Rain is an open-source distributed computational framework for processing of large-scale task-based pipelines.

Rain aims to lower the entry barrier to the world of distributed computing. Our intention is to provide a light yet robust distributed framework that features an intuitive Python API, straightforward installation and deployment with insightful monitoring on top.

Despite that this is an early release of Rain, it is a fully functional project that can be used out-of-the box. Being aware that there is still a lot that can be improved and added, we are looking for external users and collaborators to help to move this work forward. Talk to us online at Gitter or via email and let us know what your projects and use-cases need, submit bugs or feature requests at GitHub or even contribute with pull requests.

Features

  • Dataflow programming. Computation in Rain is defined as a flow graph of tasks. Tasks may be built-in functions, Python/C++/Rust code, or an external applications, short and light or long-running and heavy. The system is designed to integrate any code into a pipeline, respecting its resource requirements, and to handle very large task graphs (hundreds thousands tasks).

  • Easy to use. Rain was designed to be easy to deployed anywhere, ranging from a single node deployments to large-scale distributed systems and clouds ranging thousands of cores.

  • Rust core, Python/C++/Rust API. Rain is written in Rust for safety and efficiency and has a high-level Python API to Rain core infrastructure, and even supports Python tasks out-of-the-box. Rain also provides libraries for writing own tasks in C++ and Rust.

  • Monitoring. Rain is designed to support both online and postmortem monitoring.

    Dashboard screencast

Documentation

OverviewQuickstartUser guidePython APIExamples

Quick start

  • Download binary
$ wget https://github.com/substantic/rain/releases/download/v0.4.0/rain-v0.4.0-linux-x64.tar.xz
$ tar xvf rain-v0.4.0-linux-x64.tar.xz
  • Install Python API
$ pip3 install rain-python
  • Start server & a single local governor
$ ./rain-v0.4.0-linux-x64/rain start --simple
  • Rain "Hello world" in Python
from rain.client import Client, tasks, blob

client = Client("localhost", 7210)

with client.new_session() as session:
    task = tasks.Concat((blob("Hello "), blob("world!")))
    task.output.keep()
    session.submit()
    result = task.output.fetch().get_bytes()
    print(result)

Installation via cargo

If you have installed Rust, you can install and start Rain as follows:

$ cargo install rain_server

$ pip3 install rain-python

$ rain start --simple

Read the docs for more examples.

You can’t perform that action at this time.