Home
Jetstream is a real-time stream processing system for analyzing live event streams to enable business to react to signals much earlier than is possible with batch oriented systems like Hadoop.
It is targeted for security, risk, machine learning, fault monitoring, predictive analytics and many more use cases where there is a need to track, detect and react to user or machine behavior patterns over windows of time. Examples of these could be monitoring DOS attacks, predicting failures by watching metrics, correlating events etc.
It provides all the infrastructure to deploy, manage, monitor and flow events through the pipeline enabling easy deployment, management and monitoring of the pipeline. It provides many off the shelf components to enable applications to be built and deployed quickly. Business logic for processing the stream can be written in a SQL like langauge called EPL which can be hot deployed in to a running application.
It has several capabilities which enables building and deploying of clustered stream processing applications in a cloud environment effortless.
Jetstream offers a very simple abstraction for creating stream processing pipeline through a declarative syntax. Pipelines are created at run time and can be changed at run time without code roll out in most cases.
The pipeline consists of one or more stages of Jetstream components. Pipelines can span both within and across application boundaries. Each stage is either an Event Sink or an Event Source or both. The framework provides the following abstractions - EventSink, EventSource, Channel, Processor & Event. These are the basic building blocks of all Jetstream Applications. One can create implementations of these abstractions and wire them in to a pipeline.
Pipelines extending across application boundaries are seamlessly tied together through Jetstream’s cluster messaging which allows the apps to horizontally scale and schedule events from sources to sinks using many different message distribution patterns like weighted round robin, Weighted Random, Consistent Hashing and Modulo.
Channel abstraction enables creation of specialized channels. Channels interface with external sources and sinks of events. Channels are of two types Inbound and outbound. Inbound Channels are Event Sources for the application and interface with entities sending events to the app. They are always wired to be the first stage of a pipeline within an application boundary. Outbound Channels are Event Sinks for the application and interface with entities consuming events sent by the application. They are always wired to be the last stage of the pipeline within application boundary. The framework provides the following channel implementations Viz. Cluster messaging, Kafka and REST channels.
Processors act as both Event Sink and Event Source. They are wired in to the pipeline to serve as intermediate processing stages within an application boundary. Implementations of Processor abstraction consume the event, process it and produce another event for a downstream stage.
Jetstream has integrated with Esper – A specialized processor implementation called Esper Processor encapsulating Esper is provided by Jetstream as an off the shelf component. Esper allows users to write stream processing business logic in a SQL like language called EPL. EPLs are compiled at runtime in to java code. Jetstream allows users to apply new EPL to Esper Processor at runtime without disrupting event streams. Jetstream also has made several language extensions to EPL to integrate Esper with Spring and the pipeline. One can create new dimensions from incoming events and inject these new dimensions in to the event using these extensions.
It is built using several Open Source Technologies. It does not require any run time licenses.
Follow Jetstream Tutorial to explore Jetstream. Hope you have fun exploring and will contribute back.
- Sharad Murthy
- Xinglang Wang
- Rajeshwari Muthupandian
- Xiaoju Wu
- Robin Qiao
- Ken Wang
- Kevin Fang
- Lisa Li
- Warren Jin
- Dyutimoy Sarkar
- Parin Jogani
- Naga Mayakuntla
- Tim Robison
- George Jin
- Vijeya Anbalagan