Mirror of Apache S4
Clone or download
Failed to load latest commit information.
lib Update artifacts for release Jul 31, 2012
subprojects Update artifacts for release Jul 31, 2012
test-apps Update artifacts for release Jul 31, 2012
.gitignore Update license for related projects + removed code using LGPL Jul 25, 2012
NOTICE.txt Update license and notice Jul 27, 2012
README.md Update artifacts for release Jul 31, 2012
RELEASE_NOTES.html Fix classpath issue for adapter by adding reference paths to tools li… Jul 26, 2012
build.gradle Fix release notes suffix Jul 31, 2012
gradlew.bat Updgraded gradle to 1.0 Jul 1, 2012
settings.gradle Fix build version and dependencies in subproject files, Jul 25, 2012


Apache S4


S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

S4 0.5.0 is a complete refactoring of the previous version of S4. It grounds on the same concepts (partitioning inspired by map-reduce, actors-like distribution model), but with the following objectives:

  • cleaner and simpler API
  • robust configuration through statically defined modules
  • cleaner architecture
  • robust codebase
  • easier to develop S4 apps, to test, and to use the platform

We added the following core features:

  • TCP-based communications
  • state recovery through a flexible checkpointing mechanism
  • inter-cluster/app communications through a pub-sub model
  • dynamic application deployment
  • toolset for easily starting S4 nodes, testing, packaging, deploying and monitoring S4 apps


For the latest information about S4, please visit our website at:


and our wiki, at:


Currently the wiki contains the most relevant and up-to-date documentation.

Source code is available here: https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git


  • JDK 6 or higher
  • *nix or macosx (you may build the project and develop S4 applications with microsoft windows though, the only limitation is that the "s4" script has not been ported to windows yet)

How to build

This only applies if you checkout from the source repository or if you download a released source package.

We use gradle http://gradle.org as the build system.

  • From the root directory of the S4 project:

./gradlew install

This will build the packages and install the artifacts in the local maven repository.

  • Then, build the tools:

./gradlew s4-tools:installApp

This will build the tools so that you can drive the platform through the "s4" command.

Directory structure

  • If you have a source package:

    • root directory contains build and utility scripts, as well as informative files
    • config/ directory contains configuration files for source code formatting
    • doc/ directory contains the javadoc
    • gradle/ directory contains libraries used by the gradle build tool
    • lib/ directory contains some other gradle libraries
    • subproject/ directory contains the plaftorm subprojects: base, comm, core, tools, as well as example (example is not fully ported to 0.5.0)
    • test-apps/ directory contains some examples (some of them very trivial are used in regression tests)
  • If you have a binary package:

    • root directory contains the s4 utility script as well as informative files
    • doc/ directory contains the javadoc
    • lib/ directory contains :
      • the platform libraries (prefixed with "s4")
      • the dependencies
    • bin/ directory contains some scripts that are used by the s4 script
    • gradle/ directory contains libraries used for building S4 projects