Gearpump

Web site: http://gearpump.io

Gearpump is a lightweight real-time big data streaming engine. It is inspired by recent advances in the Akka framework and a desire to improve on existing streaming frameworks.

The name Gearpump is a reference to the engineering term “gear pump,” which is a super simple pump that consists of only two gears, but is very powerful at streaming water.

We model streaming within the Akka actor hierarchy.

Per initial benchmarks we are able to process near 18 million messages/second (100 bytes per message) with a 8ms latency on a 4-node cluster.

For steps to reproduce the performance test, please check Performance benchmark

Design Document

There is a 20 pages technical paper on typesafe blog, with technical highlights https://typesafe.com/blog/gearpump-real-time-streaming-engine-using-akka

Introduction and Motivation

Check Introduction

Getting Started

Check guide Get Started

How to Build

1). Clone the Gearpump repository

  git clone https://github.com/gearpump/gearpump.git
  cd gearpump

2). Build package

  ## Please use scala 2.11 or 2.10
  ## The target package path: output/target/gearpump-${version}.zip
  sbt clean +assembly +packArchiveZip

After the build, there will be a package file gearpump-${version}.zip generated under output/target/ folder.

To build scala document, use

   ## Will generate the scala doc under target/scala_2.xx/unidoc/
   sbt unidoc

NOTE: The build requires network connection. If you are behind an enterprise proxy, make sure you have set the proxy in your env before running the build commands. For windows:

set HTTP_PROXY=http://host:port
set HTTPS_PROXY= http://host:port

For Linux:

export HTTP_PROXY=http://host:port
export HTTPS_PROXY= http://host:port

How to run Gearpump integration test

Gearpump has an integration test system which is based on Docker.

Prerequisite of integration test

To run Gearpump integration test, we first must have Docker installed in your PATH.

And you need to have Docker well configured if you are behind firewall. You can run command docker pull stanleyxu2005/gpct-jdk8 to check whether you have Docker installed and well configured.

Set docker proxy if you behind a firewall

In ubuntu, you can modify file /etc/default/docker and enable the http_proxy setting.

Run integration test

After Docker well configured, you can run the whole integration test via command:

sbt it:test

Concepts

Check guide Concepts

How to write a Gearpump Application

Check guide Streaming Application Developer Guide

Maven dependencies

Check Maven settings

Further information

• Document site gearpump.io
• User List: gearpump-user.
• Report issues: issue tracker

Contributors (time order)

• Sean Zhong
• Kam Kasravi
• Manu Zhang
• Huafeng Wang
• Weihua Jiang
• Suneel Marthi
• Stanley Xu
• Tomasz Targonski
• Sun Kewei

Contacts:

Please use the google user list if possible. For things that are not OK to be shared in maillist, please contact: xiang.zhong@intel.com kam.d.kasravi@intel.com weihua.jiang@intel.com

License

Gearpump itself is licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 For library it used, please see /LICENSE

Acknowledgement

The netty transport code work is based on Apache Storm. Thanks Apache Storm contributors.

The cgroup code work is based on JStorm. Thanks JStorm contributors.

Thanks to Jetbrains for providing IntelliJ IDEA Free Open Source License.