Skip to content
BitSwan Pump: A real-time stream processor for Python 3.5+
Branch: master
Clone or download
mpavelka Merge branch 'mysql-sink-bulk' into 'master'
MySQLSink: use executemany; refarctor bulking

See merge request bitswan/bspump!14
Latest commit e88fb4c Apr 8, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bspump Removing obsolete code from MySQLConnection Apr 8, 2019
doc es doc Apr 4, 2019
etc
examples/data avro sink finished Nov 21, 2018
utils
.gitignore
CONTRIBUTING.md Create CONTRIBUTING.md May 1, 2018
Dockerfile Fixing example dockerfile. Oct 31, 2018
LICENSE Update LICENSE Dec 19, 2018
README.rst
bspump-amqp.py Add a UDP source (#82) Dec 13, 2018
bspump-analyzer.py Add analyzers and specifically TimeWindowAnalyzer (#62) Nov 6, 2018
bspump-avro-sink.py avro sink finished Nov 21, 2018
bspump-avro-source.py Add a support of Apache Avro format (#70) Nov 15, 2018
bspump-csv.py Enhance of the CSV example. May 25, 2018
bspump-elasticlookup.py added paging switch off Feb 25, 2019
bspump-es-es.py added comments Mar 1, 2019
bspump-es-sink.py ElasticSearch Source (#78) Nov 30, 2018
bspump-es-source.py ElasticSearch Source (#78) Nov 30, 2018
bspump-http.py Fix of the HTTP client retry when the target is completely down. Oct 25, 2018
bspump-influx.py
bspump-ipgeo.py Add a IPGeoLookup Nov 9, 2018
bspump-kafka.py Improve on the Kafka connector. Dec 6, 2018
bspump-lookup.py Add a master/slave concept for Lookups (#63) Nov 6, 2018
bspump-mongolookup.py
bspump-mysql-binary-log.py added finally Mar 11, 2019
bspump-mysql-lookup.py ElasticSearch and MySQL lookups (#100) Feb 21, 2019
bspump-mysql.py
bspump-parquet.py
bspump-pumpbuilder.py name fixes Mar 21, 2019
bspump-router.py
bspump-sa.py
bspump-slack.py Add a Slack Sink (#56) Oct 31, 2018
bspump-tcp.py Fix. Feb 14, 2019
bspump-transf.py
bspump-twa.py Separate Time Window object from an Analyzer. (#77) Nov 26, 2018
bspump-udp.py Add a UDP source (#82) Dec 13, 2018
bspump-web.py Fix. Feb 14, 2019
bspump-ws-client.py Add a WebSocket sink and source. (#80) Dec 10, 2018
bspump-ws-server.py Add a WebSocket sink and source. (#80) Dec 10, 2018
setup.py

README.rst

BSPump: A real-time stream processor for Python 3.5+

Join the chat at https://gitter.im/TeskaLabs/bspump

Principles

  • Write once, use many times
  • Schema-less
  • High performance
  • Back pressure
  • Simple to use and well documented, so anyone can write their own stream processor
  • Sliding time window
  • Asynchronous via Python 3.5+ async/await and asyncio
  • Event driven Architecture / Reactor pattern
  • Single-threaded core but compatible with threads
  • Compatible with pypy, Just-In-Time compiler capable of boosting Python code performace more then 5x times
  • Modularized

Stream processor example

#!/usr/bin/env python3
import bspump
import bspump.socket
import bspump.common
import bspump.elasticsearch

class MyPipeline(bspump.Pipeline):
    def __init__(self, app):
        super().__init__(app)
        self.build(
            bspump.socket.TCPStreamSource(app, self),
            bspump.common.JSONParserProcessor(app, self),
            bspump.elasticsearch.ElasticSearchSink(app, self, "ESConnection")
        )


if __name__ == '__main__':
    app = bspump.BSPumpApplication()
    svc = app.get_service("bspump.PumpService")
    svc.add_connection(bspump.elasticsearch.ElasticSearchConnection(app, "ESConnection"))
    svc.add_pipeline(MyPipeline(app))
    app.run()

Blank application setup

You can clone blank application from it's own repository.

Available technologies

  • bspump.amqp AMQP/RabbitMQ connection, source and sink
  • bspump.elasticsearch ElasticSearch connection, source and sink
  • bspump.file (plain files, JSON, CSV)
  • bspump.http.client HTTP client source, WebSocket client sink
  • bspump.http.web HTTP server source and sink, WebSocket server source
  • bspump.influxdb InfluxDB connection and sink
  • bspump.kafka Kafka connection, source and sink
  • bspump.mysql MySQL connection, source and sink
  • bspump.parquet Apache Parquet file sink
  • bspump.avro Apache Avro file source and sink
  • bspump.socket TCP source, UDP source
  • bspump.mongodb MongoDB connection and lookup
  • bspump.slack Slack connection and sink
  • bspump.trigger Opportunistic, PubSub and Periodic triggers
  • bspump.crypto Cryptography
    • Hashing: SHA224, SHA256, SHA384, SHA512, SHA1, MD5, BLAKE2b, BLAKE2s
    • Symmetric Encryption: AES 128, AES 192, AES 256
  • bspump.analyzer
    • Time Window analyzer
    • Time Drift analyzer
  • bspump.lookup
    • GeoIP Lookup

Google Sheet with technological compatiblity matrix: https://docs.google.com/spreadsheets/d/1L1DvSuHuhKUyZ3FEFxqEKNpSoamPH2Z1ZaFuHyageoI/edit?usp=sharing

High-level architecture

Schema of BSPump high-level achitecture

Licence

BSPump is an open-source software, available under BSD 3-Clause License.

You can’t perform that action at this time.