Clone or download
Pull request Compare This branch is 1 commit ahead, 102 commits behind apache:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
common
executor
gradle/wrapper
leader
sdk
.gitignore
.ruby-version
.travis.yml
DISCLAIMER
LICENSE
NOTICE
README.md
ama-key
ama-key.pub
build.gradle
gradlew
gradlew.bat
settings.gradle

README.md

Apache Amaterasu Build Status

                                           /\
                                          /  \ /\
                                         / /\ /  \
    _                 _                 / /  / /\ \   
   /_\   _ __   __ _ | |_  ___  _ _  __(_( _(_(_ )_) 
  / _ \ | '  \ / _` ||  _|/ -_)| '_|/ _` |(_-<| || |
 /_/ \_\|_|_|_|\__,_| \__|\___||_|  \__,_|/__/ \_,_|

Apache Amaterasu is an open-source, deployment tool for data pipelines. Amaterasu allows developers to write and easily deploy data pipelines, and clusters manage their configuration and dependencies.

Download

For this preview version, we have packaged amaterasu nicely for you to just download and extract. Once you do that, you are just a couple of easy steps away from running your first job.

Creating a dev/test Mesos cluster

We have also created a Mesos cluster you can use to test Amaterasu or use for development purposes. For more details, visit the amaterasu-vagrant repo

Configuration

Configuring amaterasu is very simple. Before running amaterasu, open the amaterasu.properties file in the top-level amaterasu directory, and verify the following properties:

property Description Default value
zk The ZooKeeper connection
string to be used by
amaterasu
192.168.33.11
master The clusters' Mesos master 192.168.33.11
user The user that will be used
to run amaterasu
root

Running a Job

To run an amaterasu job, run the following command in the top-level amaterasu directory:

ama-start.sh --repo="https://github.com/shintoio/amaterasu-job-sample.git" --branch="master" --env="test" --report="code" 

We recommend you either fork or clone the job sample repo and use that as a starting point for creating your first job.

Apache Amaterasu Developers Information

Building Apache Amaterasu

to build the amaterasu home dir (for dev purposes) run:

./gradlew buildHomeDir test

to create a distributable jar (clean creates the home dir first) run:

./gradlew buildDistribution test

Architecture

Amaterasu is an Apache Mesos framework with two levels of schedulers:

  • The ClusterScheduler manages the execution of all the jobs
  • The JobScheduler manages the flow of a job

The main clases in Amateraso are listed bellow:

+-------------------------+   +------------------------+
| ClusterScheduler        |   | Kami                   |
|                         |-->|                        |
| Manage jobs:            |   | Manages the jobs queue |
| Queue new jobs          |   | and Amaterasu cluster  |
| Reload interrupted jobs |   +------------------------+
| Monitor cluster state   |
+-------------------------+
            |
            |     +------------------------+
            |     | JobExecutor            |
            |     |                        |
            +---->| Runs the Job Scheduler |
                  | Communicates with the  |
                  | ClusterScheduler       |
                  +------------------------+
                             |
                             |
                  +------------------------+      +---------------------------+                      
                  | JobScheduler           |      | JobParser                 |
                  |                        |      |                           |
                  | Manages the execution  |----->| Parses the kami.yaml file |
                  | of the job, by getting |      | and create a JobManager   |
                  | the  execution flow    |      +---------------------------+
                  | fron the JobManager    |                    |
                  | and comunicating with  |      +---------------------------+
                  | Mesos                  |      | JobManager                |                      
                  +------------------------+      |                           |
                             |                    | Manages the jobs workflow |
                             |                    | independently of mesos    |
                  +------------------------+      +---------------------------+
                  | ActionExecutor         |
                  |                        |
                  | Executes ActionRunners |
                  | and manages state for  |
                  | the executor           |
                  +------------------------+