Netflow v9 not forwarding to Kafka when v9 element not recognized.

[vijaysar]

I am using a Cisco ASA to test this out with latest software and flow exporter in v9 format. It looks to me like the IPFIX decoder uses non-fatal error on unidentified element
--- ipfix/decoder.go lines 490-494
if !ok {
return nil, nonfatalError(fmt.Errorf("IPFIX element key (%d) not exist",
tr.FieldSpecifiers[i].ElementID))
}
-- in netflow (below) it seems like it is not a "Nonfatal Error" - causing the exporter to ignore.
-- netflow/v9/decoder.go lines 337-341
if !ok {
return nil, fmt.Errorf("Netflow element key (%d) not exist",
tr.FieldSpecifiers[i].ElementID)
}

Can you please update this and roll out to a new deb package so I can test. I have other suggestions for the software but they are not as much bugs as features I will put them in a different issue.

The setup I have is quiet simple Cisco ASA 5512X

[(outside) Cisco ASA (inside)]=>[Linux server vflow daemon]=>[Kafka]

Relevant cisco ASA configuration:
policy-map global_policy
class flow_export_class
flow-export event-type all destination 172.22.0.1
class-map flow_export_class
match access-list flow_export_vpn
access-list flow_export_vpn extended permit ip any4 xx.xx.5.0 255.255.255.0

Relevant Linux setup:
vijay@linux: more /etc/vflow/vflow.conf
netflow9-workers: 50
ipfix-tpl-cache-file: /usr/local/vflow/vflow.templates
netflow9-tpl-cache-file: /usr/local/vflow/netflow.templates
netflow9-topic: kafka.vflow.netflow
vijay@linux: more /etc/vflow/mq.conf
brokers:
- 172.22.0.1:9092
retry-max: 2
retry-backoff: 10
verify-ssl: false

When I test with vflow_stress everything is working just right, no problem. Kafka streamer is seeing the data and I can subscribe to it no problem.
...snip..{"I":152,"V":1485886990569},{"I":153,"V":1485886990569},{"I":136,"V":1},{"I":243,"V":0},{"I":245,"V":0}],[{"I":8,"V":"72.21.81.253"},{"I":12,"V":"167.21.142.42"},{"I":5,"V":0},{"I":4,"V":6},{"I":7,"V":80},{"I":11,"V":4814},{"I":32,"V":0},{"I":10,"V":939},{"I":58,"V":0},{"I":9,"V":24},{"I":13,"V":17},{"I":16,"V":4200000000},{"I":17,"V":30641},{"I":15,"V":"4.68.71.197"},{"I":6,"V":"0x10"},{"I":14,"V":1630},{"I":1,"V":7500},{"I":2,"V":5},{"I":52,"V":63},{"I":53,"V":63},{"I":152,"V":1485886571990},{"I":153,"V":1485887041099},{"I":136,"V":2},{"I":243,"V":0},{"I":245,"V":0}]]}

[mehrdadrad]

@ericavijay good catch! I'll fix this soon.

[vijaysar]

One thing I noticed if I change just the two lines in netflow/v9/decoder.go it does not seem to be enough. There is probably more changes needed to log and move on to "pass" through to the queue messages beign generated. Didnt have time to read to read the code in detail, you as experts will probably get to the bottom of it faster! Vijay

…

On Mon, Nov 6, 2017 at 11:36 AM, Mehrdad Arshad Rad < ***@***.***> wrote: @ericavijay <https://github.com/ericavijay> good catch! I'll fix this soon. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADIHlhcAE6D08dvlONXbZ0gf2QRhYAnbks5szzWLgaJpZM4QSkg_> .

[tim-hutchinson]

Seconding this, seeing it on IPFIX where there are enterprise-specific information elements sent through. I'm going to start working through the decoder tomorrow to start handling it.

With IPFIX at least, I think the output format would need to change the handle outputting the enterprise number. What are your thoughts on the best way to handle that?

// ENTERPRISE_NUMBER = 123
// ENTERPRISE_INFORMATION_ELEMENT_ID = 4567

// Stick it on the existing ID
{"I":123.4567,"V":"192.16.28.217"}

// Add a top level field
{"E":123 "I":4567,"V":"192.16.28.217"}

// Make I an object
{
    "I": {
        "E": 123, 
        "I": 4567
    },
    "V":"192.16.28.217"
}

Of those, adding a top-level field makes the most sense to me. String concat just forces consumers to add parsing logic and making I an object would break any current parsers. For non enterprise elements, E could not be serialized, or serialized as a nil/0?

[eksantrik]

Hello, has this issue been fixed? I think I am having a similar issue with my Cisco ASA Netflow v9.