Skip to content
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.
Java TypeScript HTML JavaScript CSS Shell Dockerfile
Branch: dev
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Add some exclusions to rat plugin Dec 26, 2019
.gitlab/issue_templates STREAMPIPES-24: Update license headers Dec 8, 2019
archetypes STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
development STREAMPIPES-26: Update license headers in all files Dec 9, 2019
streampipes-app-file-export STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-backend STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-code-generation STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-commons STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-config STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-connect-container-master Merge remote-tracking branch 'origin/dev' into dev Jan 9, 2020
streampipes-connect-container-worker [STREAMPIPES-17] WIP Connect Assets Jan 8, 2020
streampipes-connect-container [STREAMPIPES-17] WIP Connect Assets Jan 8, 2020
streampipes-connect [STREAMPIPES-22] manual merge from old repo Dec 18, 2019
streampipes-container-base STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-container-embedded STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-container-standalone STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-container [STREAMPIPES-17] WIP Connect Assets Jan 8, 2020
streampipes-dataformat-cbor STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-dataformat-fst STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-dataformat-json STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-dataformat-smile STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-dataformat STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-logging
streampipes-measurement-units STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-messaging-jms STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-messaging-kafka STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-messaging STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-model-client STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-model [STREAMPIPES-22] finish support for collection property Jan 8, 2020
streampipes-performance-tests STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-pipeline-management STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-rest-shared [STREAMPIPES-22] finish support for collection property Jan 8, 2020
streampipes-rest STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-sdk
streampipes-serializers STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-sources STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-storage-api STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-storage-couchdb STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-storage-management STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-storage-rdf4j STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-test-utils STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-user-management STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-vocabulary Merge remote-tracking branch 'origin/dev' into dev Jan 9, 2020
streampipes-wrapper-distributed STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-wrapper-flink STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-wrapper-kafka-streams STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-wrapper-siddhi STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-wrapper-spark STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-wrapper-standalone STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
streampipes-wrapper STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
tools/maven STREAMPIPES-2: Rename project namespace to Apache coordinates Dec 11, 2019
ui Merge branch 'feature_CollectionProperty' into dev Jan 8, 2020
.gitignore Add m2 directory to gitignore Dec 26, 2019
.gitlab-ci.yml Fix typo Dec 17, 2019
CHANGELOG.md STREAMPIPES-24: Update license headers Dec 8, 2019
DISCLAIMER-WIP STREAMPIPES-19: Add disclaimer, add preliminary (incomplete) license … Dec 14, 2019
Jenkinsfile STREAMPIPES-45: Add license header to Jenkinsfile Dec 28, 2019
LICENSE STREAMPIPES-26: Update license headers in all files Dec 9, 2019
LICENSE-binary STREAMPIPES-19: Add disclaimer, add preliminary (incomplete) license … Dec 14, 2019
NOTICE STREAMPIPES-19: Add disclaimer, add preliminary (incomplete) license … Dec 14, 2019
NOTICE-binary STREAMPIPES-19: Add disclaimer, add preliminary (incomplete) license … Dec 14, 2019
README.md Merge branch 'dev' of https://github.com/apache/incubator-streampipes… Dec 17, 2019
install.sh STREAMPIPES-24: Update license headers Dec 8, 2019
pom.xml Add some exclusions to rat plugin Dec 26, 2019

README.md

Github Actions Docker pulls Maven central License Last commit Twitter


StreamPipes Logo

Self-Service Data Analytics for the (Industrial) IoT

StreamPipes is a self-service (Industrial) IoT toolbox to enable non-technical users to connect , analyze and explore IoT data streams.

StreamPipes Pipeline
     Editor


Table of contents


About Apache StreamPipes

Apache StreamPipes (incubating) enables flexible modeling of stream processing pipelines by providing a graphical modeling editor on top of existing stream processing frameworks.

It leverages non-technical users to quickly define and execute processing pipelines based on an easily extensible toolbox of data sources, data processors and data sinks. StreamPipes has an exchangeable runtime execution layer and executes pipelines using one of the provided wrappers, e.g., for Apache Flink or Apache Kafka Streams.

Pipeline elements in StreamPipes can be installed at runtime - the built-in SDK allows to easily implement new pipeline elements according to your needs. Pipeline elements are standalone microservices that can run anywhere - centrally on your server, in a large-scale cluster or close at the edge.

Use Cases

StreamPipes allows you to connect IoT data sources using the SDK or the built-in graphical tool StreamPipes Connect.

The extensible toolbox of data processors and sinks supports use cases such as

  • Continuously store IoT data streams to third party systems (e.g., databases)
  • Filter measurements on streams (e.g., based on thresholds or value ranges)
  • Harmonize data by using data processors for transformations (e.g., by converting measurement units and data types or by aggregating measurements)
  • Detect situations that should be avoided (e.g., patterns based on time windows)
  • Wrap Machine Learning models into data processors to perform classifications or predictions on sensor and image data
  • Visualize real-time data from sensors and machines using the built-in Live Dashboard

Installation

The quickest way to run StreamPipes is the Docker-based installer script available for Unix, Mac and Windows (10).

It's easy to get started:

  1. Make sure you have Docker and Docker Compose installed.
  2. Clone or download the installer script from https://www.github.com/apache/incubator-streampipes-installer
  3. Execute ./streampipes start
  4. Enter the hostname and choose the version you'd like to run (the Lite version runs with less memory assigned to Docker (< 6 GB), use the full version if you have more memory available)
  5. Open your browser, navigate to http://YOUR_HOSTNAME_HERE and follow the installation instructions.
  6. Once finished, switch to the pipeline editor and start the interactive tour or check the online tour to learn how to create your first pipeline!

For a more in-depth manual, read the installation guide at https://streampipes.apache.org/docs/docs/user-guide-installation/!

Pipeline Elements

StreamPipes includes a repository of ready-to-use pipeline elements. A description of the standard elements can be found in the Github repository streampipes-extensions.

Extending StreamPipes

You can easily add your own data streams, processors or sinks. A Java-based SDK and several run-time wrappers for popular streaming frameworks such as Apache Flink, Apache Spark and Apache Kafka Streams (and also plain Java programs) can be used to integrate your existing processing logic into StreamPipes. Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.

Check our developer guide at https://streampipes.apache.org/docs/docs/dev-guide-introduction.

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

And don't forget to follow us on Twitter!

Contribute

We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know!

Feedback

We'd love to hear your feedback! Subscribe to users@streampipes.apache.org

License

Apache License 2.0

You can’t perform that action at this time.