PaloAlto Netflow

[mareban]

Hello,

Does someone are able to use this plugin with PaloAlto Firewall please ?

I have no problem with Cisco or software netflow exporters :) !

Thanks for your help.

https://live.paloaltonetworks.com/twzvq79624/attachments/twzvq79624/documentation_tkb/140/1/Netflow-Fields-5.0-RevA.pdf

[yvesbd]

I don't have time to address it well here now, but based on experience (and memory, which may fail me) with this plugin this year here are a few things to review. I'm planning to rewrite a version of this plugin to clean, document and fix a few other issues. But that is planned for early January.

First go look in your /var/log/td-agent/td-agent.log file
You will likely see a warning message created by this line in the parser_netflow.rb file
$log.warn "Skip unsupported field", type: type, length: length [line 370]
for the types in the templates you reference. So if you do a grep:
grep "Skip unsupported field" /var/log/td-agent/td-agent.log
You should get answers. If you look further, you'll likely see an exception and a stacktrace.
I'm not a Ruby developer, but my debugging leads me to believe that the BinData package used by this plugin has changed and that the skip method is not called correctly anymore.
If I remember what I did to fix it, there are 2 fixes possible.
One is to add the fields within the netflow_fields.yaml in the "options" section for all different type numbers you get in your logs, like:
#:

• len
• :name
  for example:
  148:
• 4
• :temp148

The other option which may or may not work, is to replace line 371 in parser_netflow.rb
from:
return [:skip, nil, {length: length}]
to:
return [[:skip, nil, {length: length}]]

Please let us know if either of these fixes works.

[mareban]

Hello,

Thanks for you reply ! Now It's OK, i just add the missing fields in template, thank you :

148:

• :uint64
• :flowId
  233:
• :string
• :firewallEvent

(the firewallEvent always blank !!!), but the plugin handle PaloAlto Netflow now Thx !

But, one last problem is to add 3 others fields from another template (Template ID 257 instead of 256) :

346:

• :uint32
• :privateEnterpriseNumber
  56701:
• :string
• :appId
  56702:
• :string
• :userId

Is the plugin able to handle this case please ?

Thanks for your help.

[yvesbd]

I didn't write the plugin, but my understanding is that all fields are stored in the same file, no matter which template they come with, so you'd have to add them to this file. To change this may require significant rewrite to the plugin. Higher numbers are left open to each manufacturer, so if you have different devices reporting, you may encounter the same # with different meaning. You'd likely need to treat it within the fluentd config (or another custom plugin). I'll post back on this thread when I have my own plugin written (I have written a few simple ones for my own use) and tested (I have some pcap files from different devices to test it out, but more tests or test files would be welcome).

[repeatedly]

I'm not familiar with PaloAlto firewall, so I don't know the differences between PaloAlto and other devices:
But you can use definitions parameter,

fluent-plugin-netflow/lib/fluent/plugin/parser_netflow.rb

Line 18 in 1fe4417

config_param :definitions, :string, default: nil

, to support missing fields. No need netflow_fields.yaml edit.

[mareban]

Hello,

Thank you all for your replies ! I've tried to had those 3 fields in the template file or in a definition file, the parsing seems correct, no more error message, but the 3 fields in not indexed in ES ???

Is the plugin able to manage 2 templates, the standard template (256) and the Enterprise template (257), Palo Alto privateEnterpriseNumber is 25461, and merge the results please ?

Thanks for your help.

346:

• :uint32
• :privateEnterpriseNumber
  56701:
• 32
• :appId
  56702:
• 64
• :userId

https://live.paloaltonetworks.com/twzvq79624/attachments/twzvq79624/documentation_tkb/140/1/Netflow-Fields-5.0-RevA.pdf

[mareban]

Hello,

I've updated to the last version and change 2 fields type and now everything is ok with app_id and user-id !

Hereis the template :

---
option:
  1:
  - 4
  - :in_bytes
  2:
  - 4
  - :in_pkts
  3:
  - 4
  - :flows
  4:
  - :uint8
  - :protocol
  5:
  - :uint8
  - :src_tos
  6:
  - :uint8
  - :tcp_flags
  7:
  - :uint16
  - :l4_src_port
  8:
  - :ip4_addr
  - :ipv4_src_addr
  9:
  - :uint8
  - :src_mask
  10:
  - 2
  - :input_snmp
  11:
  - :uint16
  - :l4_dst_port
  12:
  - :ip4_addr
  - :ipv4_dst_addr
  13:
  - :uint8
  - :dst_mask
  14:
  - 2
  - :output_snmp
  15:
  - :ip4_addr
  - :ipv4_next_hop
  16:
  - 2
  - :src_as
  17:
  - 2
  - :dst_as
  18:
  - :ip4_addr
  - :bgp_ipv4_next_hop
  19:
  - 4
  - :mul_dst_pkts
  20:
  - 4
  - :mul_dst_bytes
  21:
  - :uint32
  - :last_switched
  22:
  - :uint32
  - :first_switched
  23:
  - 4
  - :out_bytes
  24:
  - 4
  - :out_pkts
  25:
  - :uint16
  - :min_pkt_length
  26:
  - :uint16
  - :max_pkt_length
  27:
  - :ip6_addr
  - :ipv6_src_addr
  28:
  - :ip6_addr
  - :ipv6_dst_addr
  29:
  - :uint8
  - :ipv6_src_mask
  30:
  - :uint8
  - :ipv6_dst_mask
  31:
  - 3
  - :ipv6_flow_label
  32:
  - :uint16
  - :icmp_type
  33:
  - :uint8
  - :mul_igmp_type
  34:
  - :uint32
  - :sampling_interval
  35:
  - :uint8
  - :sampling_algorithm
  36:
  - :uint16
  - :flow_active_timeout
  37:
  - :uint16
  - :flow_inactive_timeout
  38:
  - :uint8
  - :engine_type
  39:
  - :uint8
  - :engine_id
  40:
  - 4
  - :total_bytes_exp
  41:
  - 4
  - :total_pkts_exp
  42:
  - 4
  - :total_flows_exp
  43:
  - :skip
  44:
  - :ip4_addr
  - :ipv4_src_prefix
  45:
  - :ip4_addr
  - :ipv4_dst_prefix
  46:
  - :uint8
  - :mpls_top_label_type
  47:
  - :uint32
  - :mpls_top_label_ip_addr
  48:
  - 1
  - :flow_sampler_id
  49:
  - :uint8
  - :flow_sampler_mode
  50:
  - :uint32
  - :flow_sampler_random_interval
  51:
  - :skip
  52:
  - :uint8
  - :min_ttl
  53:
  - :uint8
  - :max_ttl
  54:
  - :uint16
  - :ipv4_ident
  55:
  - :uint8
  - :dst_tos
  56:
  - :mac_addr
  - :in_src_mac
  57:
  - :mac_addr
  - :out_dst_mac
  58:
  - :uint16
  - :src_vlan
  59:
  - :uint16
  - :dst_vlan
  60:
  - :uint8
  - :ip_protocol_version
  61:
  - :uint8
  - :direction
  62:
  - :ip6_addr
  - :ipv6_next_hop
  63:
  - :ip6_addr
  - :bgp_ipv6_next_hop
  64:
  - :uint32
  - :ipv6_option_headers
  65:
  - :skip
  66:
  - :skip
  67:
  - :skip
  68:
  - :skip
  69:
  - :skip
  70:
  - :mpls_label
  - :mpls_label_1
  71:
  - :mpls_label
  - :mpls_label_2
  72:
  - :mpls_label
  - :mpls_label_3
  73:
  - :mpls_label
  - :mpls_label_4
  74:
  - :mpls_label
  - :mpls_label_5
  75:
  - :mpls_label
  - :mpls_label_6
  76:
  - :mpls_label
  - :mpls_label_7
  77:
  - :mpls_label
  - :mpls_label_8
  78:
  - :mpls_label
  - :mpls_label_9
  79:
  - :mpls_label
  - :mpls_label_10
  80:
  - :mac_addr
  - :in_dst_mac
  81:
  - :mac_addr
  - :out_src_mac
  82:
  - :string
  - :if_name
  83:
  - :string
  - :if_desc
  84:
  - :string
  - :sampler_name
  89:
  - :uint8
  - :forwarding_status
  91:
  - :uint8
  - :mpls_prefix_len
  95:
  - 4
  - :app_id
  148:
  - :uint64
  - :flowId
  150:
  - :uint32
  - :flowStartSeconds
  151:
  - :uint32
  - :flowEndSeconds
  152:
  - :uint64
  - :flowStartMilliseconds
  153:
  - :uint64
  - :flowEndMilliseconds
  154:
  - :uint64
  - :flowStartMicroseconds
  155:
  - :uint64
  - :flowEndMicroseconds
  156:
  - :uint64
  - :flowStartNanoseconds
  157:
  - :uint64
  - :flowEndNanoseconds
  225:
  - :ip4_addr
  - :postNATSourceIPv4Address
  226:
  - :ip4_addr
  - :postNATDestinationIPv4Address
  227:
  - :uint16
  - :postNAPTSourceTransportPort
  228:
  - :uint16
  - :postNAPTDestinationTransportPort
  233:
  - :uint8
  - :firewallEventPA
  234:
  - :uint32
  - :ingress_vrf_id
  235:
  - :uint32
  - :egress_vrf_id
  236:
  - :string
  - :vrf_name
  281:
  - :ip6_addr
  - :postNATSourceIPv6Address
  282:
  - :ip6_addr
  - :postNATDestinationIPv6Address
  346:
  - :uint32
  - :privateEnterpriseNumber
  56701:
  - :string
  - :appId
  56702:
  - :string
  - :userId

scope:
  1:
  - :ip4_addr
  - :system
  2:
  - :skip
  3:
  - :skip
  4:
  - :skip
  5:
  - :skip

Thanks for your replies and for this useful plugin.

Happy new year :)

[repeatedly]

Good to hear!

[Khodesaeed]

Hi, I am facing the same issue with Fortigate Firewalls.

Here is the log of the /var/log/td-agent/td-agent.log:

 [warn]: #0 emit transaction failed: error_class=RangeError error="bignum too big to convert into `unsigned long long'" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.14.3/lib/fluent/event.rb:60:in `write'" tag="netflow.event"

FYI, I used the template file that @mareban posted.

I appreciate any help.