Netflow filebeat 7.4.2 and template definition #14617

fredbcode · 2019-11-19T16:26:42Z

Hello,

Option -M netflow.log.custom_definitions=file.template seems inactive

With debug mode if have many "No template for ID XXX"

Eg: No template ID 263
Packet from: xxxx.xxxx.xxx.xxx:2536 src:169765857

source_id: 169765857 is in my template file
The same file works well with logstash

Another issue without debug, packets with no template are just silently dropped

fredbcode · 2019-11-19T16:51:32Z

Steps to reproduce:

-M netflow.log.custom_definitions=file.template with a template
filebeat with log level debug

The entries with source_id and id:

Eg in template:
{"825309441|263":[["ip4_addr","ipv4_src_addr"],["ip4_addr","ipv4_dst_addr"],["uint16","input_snmp"],["uint16","l4_src_port"],["uint16","l4_dst_port"],["uint8","src_tos"],["uint8","protocol"],["uint8","tcp_flags"],["ip4_addr","ipv4_next_hop"],["uint32","in_bytes"],["uint32","in_pkts"],["uint32","first_switched"],["uint32","last_switched"],["uint16","output_snmp"]]

Logs
No template ID 263
Packet from: xxxx.xxxx.xxx.xxx:2536 src:825309441

andrewkroh · 2019-11-19T23:16:44Z

The netflow module is not exposing the custom_definitions option at the current time. As a work-around you can use the netflow input directly. For example see: https://discuss.elastic.co/t/missing-user-id-field-on-netflow-from-palo-alto/208327/2?u=andrewkroh

I have opened a PR to add the missing options to the netflow module in #14628.

The exposes all of the netflow input configuration options through the netflow/log fileset. Fixes elastic#14617 Filebeat's module code was changed to allow for manifests to declare variables without default values. This means that module manifests can be written without duplicating the default values of the input types that they wrap (e.g. don't duplicate the default socket timeout value). It also changes the template evaluation to be more strict in that referencing a variable that does not exist will now cause an error instead of evaluating to "<no value>". The zeek/dhcp module referred to a variable that was not declared in its manifest. This was fixed.

fredbcode · 2019-11-21T08:55:49Z

Maybe I'm doing something wrong but it doesn't works
What do you think about add an entry in logs for requests without template ? With no log level debug there is no information at all, the entry is just silently ignored
And debug is very, very verbose

related to #14618 the cache value entry is missing ? And how it works now ? in memory ?
Maybe I'm wrong but it's like there is no cache at all, the routeur sends their template and filebeat in debug mode still complains about template

Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.281Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] FlowSet ID 263 length 156
Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.281Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] No template for ID 263
Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.282Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] Packet from:10.1.1.70:35881 src:842348289 seq:12695650
Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.282Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] Session 10.1.1.70:35881 reset (sequence=12695650 last=12695650)
Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.282Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] FlowSet ID 263 length 234
Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.282Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] No template for ID 263
Nov 21 09:09:16  test_filebeat-netflow[32702]: 2019-11-21T08:09:16.282Z#011DEBUG#011[netflow]#011netflow/input.go:77#011[netflow-v9] Packet from:10.1.1.70:35881 src:1680881921 seq:9747253

   custom_definitions:
     - /etc/filebeat/netflow_templates.cache

   "1680881921|263" : [
      [
         "ip4_addr",
         "ipv4_src_addr"
      ],
      [
         "ip4_addr",
         "ipv4_dst_addr"
      ],
      [
         "uint16",
         "input_snmp"
      ],
      [
         "uint16",
         "l4_src_port"
      ],
      [
         "uint16",
         "l4_dst_port"
      ],
      [
         "uint8",
         "src_tos"
      ],
      [
         "uint8",
         "protocol"
      ],
      [
         "uint8",
         "tcp_flags"
      ],
      [
         "ip4_addr",
         "ipv4_next_hop"
      ],
      [
         "uint32",
         "in_bytes"
      ],
      [
         "uint32",
         "in_pkts"
      ],
      [
         "uint32",
         "first_switched"
      ],
      [
         "uint32",
         "last_switched"
      ],
      [
         "uint16",
         "output_snmp"
      ]
   ],

  "842348289|263" : [
      [
         "ip4_addr",
         "ipv4_src_addr"
      ],
      [
         "ip4_addr",
         "ipv4_dst_addr"
      ],
      [
         "uint16",
         "input_snmp"
      ],
      [
         "uint16",
         "l4_src_port"
      ],
      [
         "uint16",
         "l4_dst_port"
      ],
      [
         "uint8",
         "src_tos"
      ],
      [
         "uint8",
         "protocol"
      ],
      [
         "uint8",
         "tcp_flags"
      ],
      [
         "ip4_addr",
         "ipv4_next_hop"
      ],
      [
         "uint32",
         "in_bytes"
      ],
      [
         "uint32",
         "in_pkts"
      ],
      [
         "uint32",
         "first_switched"
      ],
      [
         "uint32",
         "last_switched"
      ],
      [
         "uint16",
         "output_snmp"
      ]
   ],

fredbcode · 2019-11-21T09:11:44Z

Maybe this can be useful

In docker:

/usr/share/filebeat/bin/filebeat -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

command: filebeat -e -E output.elasticsearch.hosts=http://elasticsearch:9200 -E output.elasticsearch.worker=4 -E output.elasticsearch.bulk_max-size=12000 -E output.elasticsearch.username=logstash_netflow -E setup.ilm.rollover_alias=netflow -E setup.ilm.policy_name=netflow_policy -E setup.ilm.policy_file=/etc/filebeat/lifecycle.json -E logging.level=debug

filebeat.reference.yml

#------------------------------ NetFlow input --------------------------------
# Experimental: Config options for the Netflow/IPFIX collector over UDP input
 - type: netflow
   enabled: true
#
#  # Address where the NetFlow Collector will bind
  host: "0.0.0.0:2055"
#
#  # Maximum size of the message received over UDP
  max_message_size: 500KiB
#
#  # List of enabled protocols.
#  # Valid values are 'v1', 'v5', 'v6', 'v7', 'v8', 'v9' and 'ipfix'
#  #protocols: [ v5, v9, ipfix ]
#
#  # Expiration timeout
#  # This is the time before an idle session or unused template is expired.
#  # Only applicable to v9 and ipfix protocols. A value of zero disables expiration.
  expiration_timeout: 30m
#
#  # Queue size limits the number of netflow packets that are queued awaiting
#  # processing.
  queue_size: 3000
#
#  # Custom field definitions for NetFlow V9 / IPFIX.
#  # List of files with YAML fields definition.
  custom_definitions:
    - /etc/filebeat/netflow_templates.cache

Netflow is disable ?? There is nothing in syslog except monitoring

But with:

command: filebeat -e -E output.elasticsearch.hosts=http://elasticsearch:9200 -E output.elasticsearch.worker=4 -E output.elasticsearch.bulk_max-size=12000 -E output.elasticsearch.username=logstash_netflow -E setup.ilm.rollover_alias=netflow -E setup.ilm.policy_name=netflow_policy -E setup.ilm.policy_file=/etc/filebeat/lifecycle.json --modules netflow -M netflow.log.var.netflow_host=0.0.0.0 -M netflow.log.var.netflow_port=2055 -M netflow.log.expiration_timeout=3600m -M netflow.log.queue_size=64000

It works but template seems inactive, hundred of lines with : No template ID XXX

fredbcode · 2019-11-21T09:17:03Z

As far I can tell my problem is: The netflow module is not exposing the custom_definitions and the filebeat.reference.yml is not used (at least by netflow)

EDIT: Same result with filebeat modules enable netflow and values in netflow.yml

The exposes all of the netflow input configuration options through the netflow/log fileset. Fixes elastic#14617 Filebeat's module code was changed to allow for manifests to declare variables without default values. This means that module manifests can be written without duplicating the default values of the input types that they wrap (e.g. don't duplicate the default socket timeout value). It also changes the template evaluation to be more strict in that referencing a variable that does not exist will now cause an error instead of evaluating to "<no value>". The zeek/dhcp module referred to a variable that was not declared in its manifest. This was fixed. (cherry picked from commit e08c6ec)

fredbcode · 2019-12-03T09:12:05Z

@andrewkroh hello, are you sure that it works with 7.5.0 ?
If I try -M netflow.log.var.custom_definitions=myfile, I still have

Eg: No template ID 263
Packet from: xxxx.xxxx.xxx.xxx:2536 src:169765857

Worst if I try a wrong value, false path for custom_definitions or netflow.log.var.read_buffer=fred there is no error/warning from filebeat ?

andrewkroh · 2019-12-03T15:44:35Z

My improvement was backported after 7.5 was made so it will show up in 7.6.0. You could workaround this by using the netflow input directly rather than via the module.

) The exposes all of the netflow input configuration options through the netflow/log fileset. Fixes #14617 Filebeat's module code was changed to allow for manifests to declare variables without default values. This means that module manifests can be written without duplicating the default values of the input types that they wrap (e.g. don't duplicate the default socket timeout value). It also changes the template evaluation to be more strict in that referencing a variable that does not exist will now cause an error instead of evaluating to "<no value>". The zeek/dhcp module referred to a variable that was not declared in its manifest. This was fixed. (cherry picked from commit e08c6ec)

fredbcode mentioned this issue Nov 19, 2019

[Filebeat] Add persistent Netflow template cache #14618

Closed

andrewkroh added the Filebeat Filebeat label Nov 19, 2019

andrewkroh closed this as completed in e08c6ec Nov 21, 2019

fredbcode mentioned this issue Dec 3, 2019

7.5.0 check all parameters values for module [netflow] #14895

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Netflow filebeat 7.4.2 and template definition #14617

Netflow filebeat 7.4.2 and template definition #14617

fredbcode commented Nov 19, 2019

fredbcode commented Nov 19, 2019

andrewkroh commented Nov 19, 2019

fredbcode commented Nov 21, 2019

fredbcode commented Nov 21, 2019

fredbcode commented Nov 21, 2019 •

edited

fredbcode commented Dec 3, 2019

andrewkroh commented Dec 3, 2019

Netflow filebeat 7.4.2 and template definition #14617

Netflow filebeat 7.4.2 and template definition #14617

Comments

fredbcode commented Nov 19, 2019

fredbcode commented Nov 19, 2019

andrewkroh commented Nov 19, 2019

fredbcode commented Nov 21, 2019

fredbcode commented Nov 21, 2019

fredbcode commented Nov 21, 2019 • edited

fredbcode commented Dec 3, 2019

andrewkroh commented Dec 3, 2019

fredbcode commented Nov 21, 2019 •

edited