BSPump: A real-time stream processor for Python 3.5+
Switch branches/tags
Nothing to show
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bspump remove columns_added from _on_tick Dec 14, 2018
doc Add a tutorial. Dec 12, 2018
etc Fixing sample config. Oct 9, 2018
examples/data avro sink finished Nov 21, 2018
utils Working around missing "type" in lookup objects. Nov 26, 2018
.gitignore Ignore a local link to ASAB. Mar 10, 2018
CONTRIBUTING.md Create CONTRIBUTING.md May 1, 2018
Dockerfile Fixing example dockerfile. Oct 31, 2018
LICENSE Create LICENSE Mar 10, 2018
README.rst Add a UDP source (#82) Dec 13, 2018
bspump-amqp.py Add a UDP source (#82) Dec 13, 2018
bspump-analyzer.py Add analyzers and specifically TimeWindowAnalyzer (#62) Nov 6, 2018
bspump-avro-sink.py avro sink finished Nov 21, 2018
bspump-avro-source.py Add a support of Apache Avro format (#70) Nov 15, 2018
bspump-csv.py Enhance of the CSV example. May 25, 2018
bspump-es-sink.py ElasticSearch Source (#78) Nov 30, 2018
bspump-es-source.py ElasticSearch Source (#78) Nov 30, 2018
bspump-http.py Fix of the HTTP client retry when the target is completely down. Oct 25, 2018
bspump-influx.py Improving InfuxDB example. Mar 22, 2018
bspump-ipgeo.py Add a IPGeoLookup Nov 9, 2018
bspump-kafka.py Improve on the Kafka connector. Dec 6, 2018
bspump-lookup.py Add a master/slave concept for Lookups (#63) Nov 6, 2018
bspump-mysql.py Integrate context into bspump. #15 Apr 19, 2018
bspump-parquet.py Add a schema into a Parquet sink Oct 31, 2018
bspump-router.py Adjustment of the TeeProcessor and TeeSource. Oct 15, 2018
bspump-slack.py Add a Slack Sink (#56) Oct 31, 2018
bspump-transf.py Improve on the example. Sep 19, 2018
bspump-twa.py Separate Time Window object from an Analyzer. (#77) Nov 26, 2018
bspump-udp.py Add a UDP source (#82) Dec 13, 2018
bspump-web.py Removal of the obsolete code. Dec 1, 2018
bspump-ws-client.py Add a WebSocket sink and source. (#80) Dec 10, 2018
bspump-ws-server.py Add a WebSocket sink and source. (#80) Dec 10, 2018
setup.py Add an application banner with version number together with improveme… Nov 15, 2018

README.rst

BSPump: A real-time stream processor for Python 3.5+

Join the chat at https://gitter.im/TeskaLabs/bspump

Principles

  • Write once, use many times
  • Schema-less
  • High performance
  • Back pressure
  • Simple to use and well documented, so anyone can write their own stream processor
  • Sliding time window
  • Asynchronous via Python 3.5+ async/await and asyncio
  • Event driven Architecture / Reactor pattern
  • Single-threaded core but compatible with threads
  • Compatible with pypy, Just-In-Time compiler capable of boosting Python code performace more then 5x times
  • Modularized

Stream processor example

#!/usr/bin/env python3
import bspump
import bspump.socket
import bspump.common
import bspump.elasticsearch

class MyPipeline(bspump.Pipeline):
    def __init__(self, app):
        super().__init__(app)
        self.build(
            bspump.socket.TCPStreamSource(app, self),
            bspump.common.JSONParserProcessor(app, self),
            bspump.elasticsearch.ElasticSearchSink(app, self, "ESConnection")
        )


if __name__ == '__main__':
    app = bspump.BSPumpApplication()
    svc = app.get_service("bspump.PumpService")
    svc.add_connection(bspump.elasticsearch.ElasticSearchConnection(app, "ESConnection"))
    svc.add_pipeline(MyPipeline(app))
    app.run()

Available technologies

  • bspump.amqp AMQP/RabbitMQ connection, source and sink
  • bspump.elasticsearch ElasticSearch connection, source and sink
  • bspump.file (plain files, JSON, CSV)
  • bspump.http.client HTTP client source, WebSocket client sink
  • bspump.http.web HTTP server source and sink, WebSocket server source
  • bspump.influxdb InfluxDB connection and sink
  • bspump.kafka Kafka connection, source and sink
  • bspump.mysql MySQL connection, source and sink
  • bspump.parquet Apache Parquet file sink
  • bspump.avro Apache Avro file source and sink
  • bspump.socket TCP source, UDP source
  • bspump.mongodb MongoDB connection and lookup
  • bspump.slack Slack connection and sink
  • bspump.trigger Opportunistic, PubSub and Periodic triggers
  • bspump.crypto Cryptography
    • Hashing: SHA224, SHA256, SHA384, SHA512, SHA1, MD5, BLAKE2b, BLAKE2s
    • Symmetric Encryption: AES 128, AES 192, AES 256
  • bspump.analyzer
    • Time Window analyzer
    • Time Drift analyzer
  • bspump.lookup
    • GeoIP Lookup

High-level architecture

Schema of BSPump high-level achitecture

Licence

BSPump is an open-source software, available under BSD 3-Clause License.

BSPump is a part of Black Swan project, joint effort of Liberty Aces and TeskaLabs.