Integration of vFlow or Goflow Netflow/IPFIX collector module in a Telegraf input plugin

[door7302]

Feature Request

NetFlow/IPFIX collector as Telegraf's input plugin

Proposal:

There are several nice Go Projects that provide reliable and scalable solutions to collect Netflox/IPFIX sample information from network devices.

• vflow : https://github.com/VerizonDigital/vflow
• goflow : https://github.com/cloudflare/goflow

It would very useful to "backport" the " Netflow/IPFIX collector module" of one of this project in a dedicated Telegraf Input plugin.

Current behavior:

Desired behavior:

Use case:

Currently we collect traffic stats of our networking devices via the gnmi Telegraf plugin. We also collect in // Netflow/IPFIX with another tool and we wish to have a single collector solution to collect all data from our network. We love Telegraf solution and moreover we have already developed several Telegraf Processor Plugins to manipulate and enrich our data collected by telegraf input plugin and we wish to reuse them for Netflow/IPFIX sampled data.

[sjwang90]

Currently we don't support binary (ex: protobuf) formats. There is a PR #3421 for support that may need some more progress.

I've opened some issues in goflow/vFlow repos to see if they're willing to support any other output formats in the meantime.

• Support output formats other than protobuf over Kafka Edgio/vflow#137
• Support output formats other than protobuf over Kafka cloudflare/goflow#88

[lspgn]

GoFlow developer/maintainer here:
I'm not familiar with Telegraf/Influx, but if we are dealing with time-series, you will need to do an aggregation before insertion (eg: summing all sFlow packets by a key like destination ASN, port and protocol) which unfortunately is out of scope of the decoder.

Alternative that I've seen used with Logstash, you can have GoFlow output the samples as JSON in stdout (not using Kafka+protobuf).

[door7302]

Thank you for your interest of my request. From my side I will have a look if I could 'POC' some stuff to integrate a part of goflow code in an input plugin. I will share my tests. We will also need a plugin of "aggregation" to aggregate sample data before sending to "Timeseries" DB to avoid the explosion of cardinality :)

It's a nice challange I guess. As i'm not a "expert" in dev I will share my tests, sample codes and maybe it will help to integrate something more reliable latter :)

[sspaink]

@door7302 were you able to make any progress on the proof of concept?

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!