netflow v9 some templates not being parsed - Multiple errors: can not read the data #42
Comments
|
I think I've worked this one out.. The detector for padding between flowsets broken in vflow/netflow/v9/decoder.go:decodeSet(). It failed to detect the padding, got out-of-sync and started parsing the padding as the next record. .. Once it started eating rubbish, the promotion to an unsigned was wrapping and trying to parse 65K of data (uint16(-1)) .. Currently: The "d.reader.Len() > 4" would detect the total bytes remaining in the packet, however we also need to test for bytes remaining in the current flowset (against the header length vs the current offset). Someone will need to check that logic - I don't know Go or Netflow well. |
|
Actuallly.. there is a bit of code further down that detects the padding, but it does so by entering the loop again and peeking at the data - if there's zeros there, it bails out.. (rather than checking the current offset position in the FlowSet).. and it only does this for FlowSetID of 0 or 1 (ie templates, not data). In the packet above, FlowSetID 256 (data) has padding, so it won't get caught by this test.. The IPFIX code uses the same algorithm to peak past the end and hope for zeros. Is there some magic reason why it's done this way? |
|
|
@jgc234 thank you for looking at this issue. I tried netflow during development by nprobe as I don't have any netflow device. I can work on this issue next week. |
|
@mehrdadrad happy to help.. you could spin up a router in a VM (Juniper vSRX or Cisco CSR100v). or replay some captured packets from someone else. I haven't read the IPFIX to see how it'd different from V9.. I'm unsure if the same padding problem could occur there to. |
|
@jgc234 can you send me a pcap including templates and samples to my email address: arshad.rad@gmail.com ? |
|
@jgc234 pls try and see how it works w/ your SRX. it changed same as you mentioned. |
|
@mehrdadrad - tested and works fine for the last few hours using an SRX with netflow v9 IPv4 only.. removed netflowv9.tempaltes cache file to force correct parsing of transmitted template definitions. |
I'm having trouble with some v9 templates not being parsed from a Juniper SRX.. Some are, and some aren't.. As an example below - template id# 261 seems to fail to be defined, even though its definition gets transmitted 60 seconds by the sending router. First some background:
Router config:
From the vflow.log file.. The "can not read data" appears every minute.
(After a few days it also crashes with a backtrace that I haven't really had a look at yet)
To me it looks like the definition of template 261 is being sent every minute, along with a few flows using the same template id.. Here's a packet.
Some other templates are working fine.. I haven't worked out what the relationship is between the successful and failing ones yet.
[vflow] 2017/12/03 14:52:29 {"AgentID":"10.232.4.5","Header":{"Version":9,"Count":7,"SysUpTime":1708483,"UNIXSecs":1512272852,"SeqNum":427,"SrcID":142},"DataSets":[[{"I":35,"V":2},{"I":34,"V":1},{"I":1,"V":"0x"}],[{"I":35,"V":0},{"I":34,"V":0},{"I":1,"V":"0x"}],[{"I":35,"V":1},.......Any thoughts on how to debug this futher?
The text was updated successfully, but these errors were encountered: