Lightweight real-time big data streaming engine over Akka
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
conf GEARPUMP-11, fix code style Apr 11, 2016
core/src GEARPUMP-11, fix code style Apr 11, 2016
daemon/src GEARPUMP-11, fix code style Apr 11, 2016
docs GEARPUMP-11, fix code style Apr 11, 2016
examples GEARPUMP-11, fix code style Apr 11, 2016
experiments GEARPUMP-11, fix code style Apr 11, 2016
external GEARPUMP-11, fix code style Apr 11, 2016
integrationtest GEARPUMP-11, fix code style Apr 11, 2016
notes GEARPUMP-11, fix code style Apr 11, 2016
project GEARPUMP-11, fix code style Apr 11, 2016
services GEARPUMP-11, fix code style Apr 11, 2016
streaming/src GEARPUMP-11, fix code style Apr 11, 2016
unmanagedlibs GEARPUMP-11, fix code style Apr 11, 2016
yarnconf GEARPUMP-11, fix code style Apr 11, 2016
.coveragerc fix #132 add codecov config Dec 25, 2014
.gitignore GEARPUMP-8, fix "two machines can possibly have same worker id for si… Apr 2, 2016
.sbtopts GEARPUMP-9, Clean and fix integration test Apr 4, 2016
.travis.yml GEARPUMP-11, fix code style Apr 11, 2016
CHANGELOG.md GEARPUMP-11, fix code style Apr 11, 2016
CONTRIBUTING.md GEARPUMP-11, fix code style Apr 11, 2016
LICENSE GEARPUMP-11, fix code style Apr 11, 2016
README.md Fix REAMDE.md format Apr 20, 2017
ReleaseProcess.md GEARPUMP-11, fix code style Apr 11, 2016
jvm.sbt GEARPUMP-11, fix code style Apr 11, 2016
pubring.asc fix #359, Allow travis to auto publish signed artifacts to nexus sona… Jan 27, 2015
scalastyle-config.xml GEARPUMP-11, fix code style Apr 11, 2016
secring.asc.enc fix #359, Allow travis to auto publish signed artifacts to nexus sona… Jan 27, 2015
version.sbt GEARPUMP-11, fix code style Apr 11, 2016

README.md

Gearpump has moved to Apache. The official Gearpump repo is now hosted by Apache and mirrored to https://github.com/apache/incubator-gearpump. Issues are tracked at https://issues.apache.org/jira/browse/GEARPUMP


Gearpump GitHub release GitHub license Build Status codecov.io

Online Demo Site: http://demo.gearpump.io/

download

Gearpump is a lightweight real-time big data streaming engine. It is inspired by recent advances in the Akka framework and a desire to improve on existing streaming frameworks.

The name Gearpump is a reference to the engineering term "gear pump", which is a super simple pump that consists of only two gears, but is very powerful at streaming water.

We model streaming within the Akka actor hierarchy.

Per initial benchmarks we are able to process near 18 million messages/second (100 bytes per message) with a 8ms latency on a 4-node cluster.

For steps to reproduce the performance test, please check Performance benchmark.

Useful Resources

How to Build

1). Clone the Gearpump repository

  git clone https://github.com/gearpump/gearpump.git
  cd gearpump

2). Build package

  ## Please use scala 2.11 or 2.10
  ## The target package path: output/target/gearpump-${version}.zip
  sbt clean +assembly +packArchiveZip

After the build, there will be a package file gearpump-${version}.zip generated under output/target/ folder.

To build scala document, use

   ## Will generate the scala doc under target/scala_2.xx/unidoc/
   sbt unidoc

NOTE: The build requires network connection. If you are behind an enterprise proxy, make sure you have set the proxy in your env before running the build commands. For windows:

set HTTP_PROXY=http://host:port
set HTTPS_PROXY= http://host:port

For Linux:

export HTTP_PROXY=http://host:port
export HTTPS_PROXY= http://host:port

How to run Gearpump integration test

Gearpump has an integration test system which is based on Docker. Please check the instructions.

How to do style check before submitting a pull request?

Before submitting a PR, you should always run style check first:

  ## Run style check for compile, test, and integration test.
  sbt scalastyle test:scalastyle it:scalastyle

Contributors (time order)

Contacts

Please use the google user list if possible. For things that are not OK to be shared in mailing list, please contact: Sean Zhong, Kam Kasravi or Weihua Jiang.

License

Gearpump itself is licensed under the Apache License (2.0). For library it used, please see LICENSE.

Acknowledgement

The netty transport code work is based on Apache Storm. Thanks Apache Storm contributors.

The cgroup code work is based on JStorm. Thanks JStorm contributors.

Thanks to Jetbrains for providing a IntelliJ IDEA Free Open Source License.