not all fields filled

[timmernet]

1st of all thank you for all of your work you put into this. I am just starting out to get some netflow information from our company firewall, a Cisco ASA 5515. I've followed your guide and am getting data into elasticsearch/kibana, but a lot of fields remain emty, for example netflow.bytes, netflow.first_switched and netflow.last_switched. Logstash generating a lot of warnings: [2017-08-07T08:34:49,804][WARN ][logstash.codecs.netflow ] Template length doesn't fit cleanly into flowset {:template_id=>263, :template_length=>86, :record_length=>64}

[robcowart]

Which version of Logstash, or perhaps more important, which version of the Logstash codec are you using? I have found the codec handles a lot of scenarios that you may encounter, but sometimes vendors will send data structured in an unexpected manner, which appears to be what you are seeing. You will need to open an issue for the Netflow codec (https://github.com/logstash-plugins/logstash-codec-netflow), and ideally provide a PCAP of the flows so that it can be investigated in detail. I am also happy to test such a PCAP myself if you can provide it.

[timmernet]

I have opened an issue as you recommended. logstash-plugins/logstash-codec-netflow#87 and hopefully provided enough information to troubleshoot. Thank you

[joriws]

This OP's error is originating from Logstash's bad feature. UDP-layer do not send metadata to codec, meaning codec cannot analyze netflow-v9 template per host. So every template with same template ID will overwrite the template in logstash's memory.

Router A template 256 = xxx.yyy.zzz
Router B template 256 = aaa.bbb.ccc

So netflow will try to interpret netflow packet from A with later template received from B. Router B's netflow packet will be decoded correct unless A sends new upgraded template and then B's flows cause errors in the log described.

But above to be problem you need multiple devices sending netflow v9 templates. Do you have many firewalls?

To fix this you need to patch udp.rb to include "metadata" configuration option and capability to send this metadata upwards. Netflow-codec already understands it on code-level so only udp.rb and conf.d file needs to be fixed.

My elastiflow 10_input.logstash.conf
input {
udp {
type => "netflow"
metadata => true
port => 2055
codec => netflow {
versions => [5,9]
cache_save_path => "/var/lib/logstash/netflow_cache"
}
}
}

[robcowart]

Resolved by #29 and c6d01da