Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Mirror of Apache S4
Java Groovy CSS Shell Ruby

This branch is 126 commits behind master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
config
etc
gradle/wrapper
lib
subprojects
test-apps
.gitignore
NOTES.md
NOTICE.txt
README.md
build.gradle
gradlew
gradlew.bat
s4
settings.gradle

README.md

S4-Piper Design

The goal of this project is to incorporate feedback from various users and create a prototype with a cleaner, simpler API. The base classes are the foundation on top of which higher level layers and tools can be built. For example, one could write a domain specific language to create the application graph. Look at this as the foundation and not as the end product.

Here is a list of ideas:

  • Eliminate string identifiers completely and make it work nicely with Guice.
  • Take advantage of Java strong typing using generics in the framework. (But keep it simple for the app developer.)
  • Use dependency injection, no more "new".
  • Limit property files to simple parameters. Config file should not be a programming language. (No more XML.)
  • Configure the app in a GuiceModule.
  • Make fields final private whenever possible and use constructors instead of setter methods.
  • Make PEs as immutable as possible.
  • Hide details about partitioning from app developer. The idea is to create a graph for the prototypes and have the base classes deal with the distributed processing details.
  • Use osgi to create s4 app bundles that can be dropped in a directory and get loaded by s4. (This will take a bit longer to do.)

Please post feedback and suggestions in the s4 forum.

The repo is here:

https://github.com/leoneu/s4-piper

In this version:

  • Everything is a POJO.
  • Key framework classes (io.s4) are:
    • Event: all messages sent between PEs inherit from Event.
    • ProcessingElement: base class for PEs.
    • Stream: takes an Event subclass as the parameter type for the stream. A stream holds references to the target PEs. Streams are the edges between PEs.
    • Key: is a helper class that can return the value of a key for a specific type of event. For each type of event, one will create a KeyFinder class that knows how to get the value from the Event class.
    • App: the container for the app.

Note that when we create a PE in App, we are really creating a prototype of the PE. The actual PE instances get created every time a node receives an event that is dispatched to a specific PE prototype. The first PE has Map that holds the instances by a specific key. PE instances are clones which means that we don't call the constructor. Instead we provide an init method. Also, the instance variables are cloned. This means that references are also copied. If you wanted to have a List per PE instance, you would have to create it in the init method. This design pattern, makes thing very simple and efficient but has the downside that programmers have to understand how instances are created via clone.

For now we identify PEs using a composite string produced by KeyFinder. We must also pass the raw data types used to create the composite key. Need to think how to do this with type safety in mind.

Examples:

  • io.s4.example.counter - Generates dummy events that are counted using various keys.
  • io.s4.example.model - Training and testing a probabilistic classifier.
Something went wrong with that request. Please try again.