Repository of ready-to-use pipeline elements for StreamPipes
Branch: dev
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
streampipes-pipeline-elements-shared
streampipes-processors-aggregation-flink
streampipes-processors-enricher-flink
streampipes-processors-filters-jvm
streampipes-processors-filters-siddhi
streampipes-processors-geo-flink [RELEASE] [skip-ci]Updating develop poms back to pre merge state Nov 27, 2018
streampipes-processors-geo-jvm
streampipes-processors-image-processing-jvm
streampipes-processors-pattern-detection-flink
streampipes-processors-statistics-flink
streampipes-processors-text-mining-flink
streampipes-processors-transformation-flink
streampipes-processors-transformation-jvm Fix #75, removing unused configuration parameters Dec 1, 2018
streampipes-sinks-brokers-jvm
streampipes-sinks-databases-flink
streampipes-sinks-databases-jvm
streampipes-sinks-internal-jvm
streampipes-sinks-notifications-jvm
streampipes-sources-random-data-generator
streampipes-sources-vehicle-simulator Fix #75, removing unused configuration parameters Dec 1, 2018
streampipes-sources-watertank-simulator
.gitignore
.gitlab-ci.yml
LICENSE
README.md
pom.xml

README.md

StreamPipes

StreamPipes enables flexible modeling of stream processing pipelines by providing a graphical modeling editor on top of existing stream processing frameworks.

It leverages non-technical users to quickly define and execute processing pipelines based on an easily extensible toolbox of data sources, data processors and data sinks.

Learn more about StreamPipes at https://www.streampipes.org/

Read the full documentation at https://docs.streampipes.org

StreamPipes Pipeline Elements

This project provides a library of several pipeline elements that can be used within the StreamPipes toolbox.

Currently, the following pipeline elements are available:

Data Sources

  • Watertank Simulator: A data simulator that replays data from an Industrial IoT use case. Several streams are provided such as flow rate, water level and more.
  • Vehicle Simulator: Provides a simulated stream that replays location-based real-time data of a vehicle.
  • Random Data Generator: Several streams that produce randomly generated data in an endless stream.

Data Processors

  • Aggregation
  • Count Aggregation
  • Event Rate
  • Timestamp Enricher
  • Numerical Filter: Filters sensor values based on a configurable threshold value.
  • Text Filter: Filters text-based fields by a given string value.
  • Projection: Outputs a configurable subset of the fields available in an input event stream.
  • Geocoding
  • Google Routing
  • Increase
  • Peak Detection
  • Statistics Summary
  • Statistics Summary Window-Based
  • Field Converter
  • Field Hasher
  • Field Mapper
  • Measurement Unit Converter
  • Field Renamer

Data Sinks

  • CouchDB: Stores events in an Apache CouchDB database.
  • Dashboard: Can be used to display pipeline results in the real-time dashboard of the StreamPipes UI.
  • Kafka Publisher: Publishes events to an Apache Kafka broker.
  • Notification: Can be used to generate notifications that are shown in the notification center of the StreamPipes UI.
  • JMS
  • RabbitMQ
  • Elasticsearch
  • Dashboard
  • Notification
  • Email Notification
  • Slack Notification
  • OneSignal Notification

Contact us if you are missing some pipeline elements!

Getting started

All modules contain a docker-compose template in the deployment folder of each module. Copy this into your StreamPipes docker-compose file (see the docs for more info) and start StreamPipes! All pipeline elements are automatically registered (but do not forget to install them under Pipeline Element Installation).

It's easy to get started:

Feedback

We'd love to hear your feedback! Contact us at feedback@streampipes.org