bwNetFlow - Open Source Network Flow Analysis Suite
The bwNetFlow suite uses existing software and provides glue codes to allow for large scale network flow analysis.
We provide a set of tools working with Kafka as consumers and/or producers to establish a flow monitoring analysis pipeline. While these tools can be combined in any specific way, the core components and wiring in our use case is as follows:
- GoFlow: Cloudflare's GoFlow receives NetFlow and produces protobuf messages in Kafka topic input
- Enricher: reads from Kafka topic input, adds domain specific knowlege (customer, direction, device info, etc), and writes protobuf messages in Kafka topic enriched
- Splitter: reads from Kafka topic enriched, writes into customer specific topics enriched-$cid for each enabled customer
- Dashboard: reads from Kafka topic enriched, aggregates the flows to counters and writes these counter values to a Time Series Database (InfluxDB or Prometheus)
The tools work with Protobuf messages for representing NetFlow packets from GoFlow - yet with an extended protobuf message as soon as enriched by the enricher component.
- consumers_example: tbd
- consumer_dumper: tbd
- consumer_counter: tbd
- outliers-detection: tbd
- protobuf_to_netflow_converter: Converts all incoming protobuf decoded messages into NetFlow v9 compliant messages.
- processor_reducer: tbd
To develop Kafka consumers/producers with Go the kafkaconnector library abstracts most of the recurrent code fragments. To develop Kafka consumers/producers with C++ the cpp_kafkaconnector library abstracts most of the recurrent code fragments.
For deploying the bwNetFlow suite we provide Ansible scripts or Docker / Docker-Compose description.