Skip to content
BitSwan Pump: A real-time stream processor for Python 3.5+
Branch: master
Clone or download
mpavelka Merge branch 'mysql-sink-bulk' into 'master'
MySQLSink: use executemany; refarctor bulking

See merge request bitswan/bspump!14
Latest commit e88fb4c Apr 8, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
bspump Removing obsolete code from MySQLConnection Apr 8, 2019
doc es doc Apr 4, 2019
examples/data avro sink finished Nov 21, 2018
.gitignore Create May 1, 2018
Dockerfile Fixing example dockerfile. Oct 31, 2018
LICENSE Update LICENSE Dec 19, 2018
README.rst Add a UDP source (#82) Dec 13, 2018 Add analyzers and specifically TimeWindowAnalyzer (#62) Nov 6, 2018 avro sink finished Nov 21, 2018 Add a support of Apache Avro format (#70) Nov 15, 2018 Enhance of the CSV example. May 25, 2018 added paging switch off Feb 25, 2019 added comments Mar 1, 2019 ElasticSearch Source (#78) Nov 30, 2018 ElasticSearch Source (#78) Nov 30, 2018 Fix of the HTTP client retry when the target is completely down. Oct 25, 2018 Add a IPGeoLookup Nov 9, 2018 Improve on the Kafka connector. Dec 6, 2018 Add a master/slave concept for Lookups (#63) Nov 6, 2018 added finally Mar 11, 2019 ElasticSearch and MySQL lookups (#100) Feb 21, 2019 name fixes Mar 21, 2019 Add a Slack Sink (#56) Oct 31, 2018 Fix. Feb 14, 2019 Separate Time Window object from an Analyzer. (#77) Nov 26, 2018 Add a UDP source (#82) Dec 13, 2018 Fix. Feb 14, 2019 Add a WebSocket sink and source. (#80) Dec 10, 2018 Add a WebSocket sink and source. (#80) Dec 10, 2018


BSPump: A real-time stream processor for Python 3.5+

Join the chat at


  • Write once, use many times
  • Schema-less
  • High performance
  • Back pressure
  • Simple to use and well documented, so anyone can write their own stream processor
  • Sliding time window
  • Asynchronous via Python 3.5+ async/await and asyncio
  • Event driven Architecture / Reactor pattern
  • Single-threaded core but compatible with threads
  • Compatible with pypy, Just-In-Time compiler capable of boosting Python code performace more then 5x times
  • Modularized

Stream processor example

#!/usr/bin/env python3
import bspump
import bspump.socket
import bspump.common
import bspump.elasticsearch

class MyPipeline(bspump.Pipeline):
    def __init__(self, app):
            bspump.socket.TCPStreamSource(app, self),
            bspump.common.JSONParserProcessor(app, self),
            bspump.elasticsearch.ElasticSearchSink(app, self, "ESConnection")

if __name__ == '__main__':
    app = bspump.BSPumpApplication()
    svc = app.get_service("bspump.PumpService")
    svc.add_connection(bspump.elasticsearch.ElasticSearchConnection(app, "ESConnection"))

Blank application setup

You can clone blank application from it's own repository.

Available technologies

  • bspump.amqp AMQP/RabbitMQ connection, source and sink
  • bspump.elasticsearch ElasticSearch connection, source and sink
  • bspump.file (plain files, JSON, CSV)
  • bspump.http.client HTTP client source, WebSocket client sink
  • bspump.http.web HTTP server source and sink, WebSocket server source
  • bspump.influxdb InfluxDB connection and sink
  • bspump.kafka Kafka connection, source and sink
  • bspump.mysql MySQL connection, source and sink
  • bspump.parquet Apache Parquet file sink
  • bspump.avro Apache Avro file source and sink
  • bspump.socket TCP source, UDP source
  • bspump.mongodb MongoDB connection and lookup
  • bspump.slack Slack connection and sink
  • bspump.trigger Opportunistic, PubSub and Periodic triggers
  • bspump.crypto Cryptography
    • Hashing: SHA224, SHA256, SHA384, SHA512, SHA1, MD5, BLAKE2b, BLAKE2s
    • Symmetric Encryption: AES 128, AES 192, AES 256
  • bspump.analyzer
    • Time Window analyzer
    • Time Drift analyzer
  • bspump.lookup
    • GeoIP Lookup

Google Sheet with technological compatiblity matrix:

High-level architecture

Schema of BSPump high-level achitecture


BSPump is an open-source software, available under BSD 3-Clause License.

You can’t perform that action at this time.