A Kafka aggregator based on the Faust Python Stream Processing library.
kafka-aggregator development is based on the Safir application template.
kafka-aggregator uses Faust's windowing feature to aggregate a stream of messages from Kafka.
kafka-aggregator implements a Faust agent, a "stream processor", that adds messages from a source topic into a Faust table. The table is configured as a tumbling window with a size, representing the window duration (time interval) and an expiration time, which specifies the duration for which the data allocated to each window will be stored. Every time a window expires, a callback function is called to aggregate the messages allocated to that window. The size of the window controls the frequency of the aggregated stream.
kafka-aggregator uses faust-avro to add Avro serialization and Schema Registry support to Faust. faust-avro can parse Faust models into Avro Schemas.
See the docs for more information.