sFlow v5 header ethertype encoding issues on Cisco ASR-9903 with IOS-XR 7.9.2

[pavel-odintsov]

Hello!

We got very curious sFlow packet dump from customer which FastNetMon was not able to parse.

Platform was following: Cisco ASR-9903 with IOS-XR 7.9.2

Configuration:

flow exporter-map SFLOW-EXPORTER-MAP-IPV4
dfbit set
packet-length 1400
version sflow v5
!
transport udp 6343
source-address y.y.y.y
source Loopback0
destination x.x.x.x
!
flow monitor-map SFLOW-MONITOR-MAP
record sflow

sflow options
  input ifindex physical
  output ifindex physical
  if-counters polling-interval 20
  extended-router
  extended-gateway
!

exporter SFLOW-EXPORTER-MAP-IPV4
!
sampler-map USS-1000
random 1 out-of 1000
!

FastNetMon's parser error was following:

2024-02-02 12:39:13,491 [DEBUG] sflow: Cannot parse nested packet using ng parser: unknown_ethertype
2024-02-02 12:39:13,491 [DEBUG] sflow: Cannot parse nested packet using ng parser: unknown_ethertype
2024-02-02 12:39:13,491 [DEBUG] sflow: Cannot parse nested packet using ng parser: unknown_ethertype

In details it was more curious that ethertype was different all the time and we found 1170 unique ethertypes:

./fastnetmon_pcap_reader sflow ~/sflow_lol_xr_data.pcap |grep Ethertype|awk '{print $2}'| head -n 10
613c
217d
197d
2178
1138
2179
213e
11fb
213d
113c

Total number:

./fastnetmon_pcap_reader sflow ~/sflow_lol_xr_data.pcap |grep Ethertype|awk '{print $2}'|sort -g | uniq |wc -l 
1170

How it looks in Wireshark:

What even more curious that we noticed that right after this obviously incorrect ethertype proper IPv4 header follows.

We slightly altered our parser to print frist byte which follow after ethertype and got following results:

./fastnetmon_pcap_reader sflow ~/sflow_lol_xr_data.pcap |grep Version|sort | uniq -c
 214835 Version: 4
     26 Version: 6

So it clearly confirms that remaining part of packet header is intact and our parser was able to parse it with ethertype value set to IPv4 manually:

{"agent_address":"0.0.0.0","destination_asn":0,"destination_country":"","destination_ip":"xx","destination_mac":"f4:xx:91","destination_port":53,"forwarding_status":"unknown","fragmentation":false,"input_interface":123,"ip_length":66,"ip_version":"ipv4","length":0,"output_interface":116,"packets":1,"protocol":"udp","sample_ratio":1000,"source_asn":0,"source_country":"","source_ip":"xx","source_mac":"44:xx:de","source_port":16681,"timestamp":0,"ttl":252}

Finally: Cisco XR has some bug which corrupts byte which stores ethertype and replaces it by some other value which appears to be random.

[pavel-odintsov]

It's actually worth to mention that this data passed through Fortinet Firewall and it may have potentially altered it.