Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sampling logic applied to un-sampled records #212

Closed
ngner opened this issue Mar 26, 2020 · 5 comments
Closed

Sampling logic applied to un-sampled records #212

ngner opened this issue Mar 26, 2020 · 5 comments

Comments

@ngner
Copy link

ngner commented Mar 26, 2020

NFDump is processing UnSampled Records as sampled.

This NFDUMP output shows "Sampled" flag set despite the NF record referring to an unsampled template.
The result is incorrect statistics being calculated, in this case 0 bytes.

Note: the device is sending several Netflow templates, some are sampled and some not.

Flow Record:
  Flags        =              0x86 FLOW, Sampled
  label        =            <none>
  export sysid =                 8
  size         =                92
  first        =        1584069206 [2020-03-13 14:13:26]
  last         =        1584069209 [2020-03-13 14:13:29]
  msec_first   =               870
  msec_last    =               870
  src addr     =    xx.xx.xx.xx
  dst addr     =    yy.yy.yy.yy
  src port     =               179
  dst port     =             54364
  fwd status   =               195
  tcp flags    =              0x00 ......
  proto        =                 6 TCP  
  (src)tos     =                 0
  (in)packets  =                 0
  (in)bytes    =                 0
  input        =                 0
  output       =               724
  out packets  =                 0
  out bytes    =                 0
  ip router    =     aa.aa.aa.aa
  engine type  =                 0
  engine ID    =                 0
  received at  =     1584069218003 [2020-03-13 14:13:38.003]

Below is the
Data Record for the above flow as Decoded in Wireshark for the above packet

FlowSet 1 [id=258] (1 flows)
FlowSet Id: (Data) (258)
FlowSet Length: 72
[Template Frame: 19 (received after this frame)]
Flow 1
Octets: 40
Post Octets: 40
Packets: 1
Post Packets: 1
[Duration: 3.000000000 seconds (switched)]
StartTime: 3583956.452000000 seconds
EndTime: 3583959.452000000 seconds
SrcPort: 179
DstPort: 54364
InputInt: 0
OutputInt: 724
Protocol: TCP (6)
Post Ip Diff Serv Code Point: 255
Classification Engine ID: PANA-L7-PEN (20)
Selector ID: 0000304400000000
Unknown Field Type: Type 66: Value (hex bytes): 00 00 00 00
Unknown Field Type: Type 65: Value (hex bytes): 0e 0c
Forwarding Status
11.. .... = ForwardingStatus: Consume (3)
..00 0011 = ForwardingStatusConsumeCode: Terminate For us (3)
Flow End Reason: End of Flow detected (3)
SrcAddr: xx.xx.xx.xx (xx.xx.xx.xx)
DstAddr: yy.yy.yy.yy (yy.yy.yy.yy)
Padding: 00

Below is the
Template Record for the above (template 258) for the above Data Record
Note it does NOT include Fields 34 or 35.

FlowSet 1 [id=0] (Data Template): 258,260,262,266,263,267,259,261,264,268,265,269
        FlowSet Id: Data Template (V9) (0)
        FlowSet Length: 1044
        Template (Id = 258, Count = 19)
            Template Id: 258
            Field Count: 19
            Field (1/19): BYTES
                Type: BYTES (1)
                Length: 8
            Field (2/19): OUT_BYTES
                Type: OUT_BYTES (23)
                Length: 8
            Field (3/19): PKTS
                Type: PKTS (2)
                Length: 4
            Field (4/19): OUT_PKTS
                Type: OUT_PKTS (24)
                Length: 4
            Field (5/19): FIRST_SWITCHED
                Type: FIRST_SWITCHED (22)
                Length: 4
            Field (6/19): LAST_SWITCHED
                Type: LAST_SWITCHED (21)
                Length: 4
            Field (7/19): L4_SRC_PORT
                Type: L4_SRC_PORT (7)
                Length: 2
            Field (8/19): L4_DST_PORT
                Type: L4_DST_PORT (11)
                Length: 2
            Field (9/19): INPUT_SNMP
                Type: INPUT_SNMP (10)
                Length: 2
            Field (10/19): OUTPUT_SNMP
                Type: OUTPUT_SNMP (14)
                Length: 2
            Field (11/19): PROTOCOL
                Type: PROTOCOL (4)
                Length: 1
            Field (12/19): postIpDiffServCodePoint
                Type: postIpDiffServCodePoint (98)
                Length: 1
            Field (13/19): APPLICATION_ID
                Type: APPLICATION_ID (95)
                Length: 9
            Field (14/19): Unknown(66)
                Type: Unknown (66)
                Length: 4
            Field (15/19): Unknown(65)
                Type: Unknown (65)
                Length: 2
            Field (16/19): FORWARDING_STATUS
                Type: FORWARDING_STATUS (89)
                Length: 1
            Field (17/19): flowEndReason
                Type: flowEndReason (136)
                Length: 1
            Field (18/19): IP_SRC_ADDR
                Type: IP_SRC_ADDR (8)
                Length: 4
            Field (19/19): IP_DST_ADDR
                Type: IP_DST_ADDR (12)
                Length: 4

We also have interspersed with the above templates
From the same router and in the same template packets the following Template record which does include the 34 and 35 Fields.

FlowSet 2 [id=1] (Options Template): 256
    FlowSet Id: Options Template(V9) (1)
    FlowSet Length: 44
    Options Template (Id = 256) (Scope Count = 1; Data Count = 7)
        Template Id: 256
        Option Scope Length: 4
        Option Length: 28
        Field (1/1) [Scope]: System
            Scope Type: System (1)
            Length: 2
        Field (1/7): TOTAL_BYTES_EXP
            Type: TOTAL_BYTES_EXP (40)
            Length: 8
        Field (2/7): TOTAL_PKTS_EXP
            Type: TOTAL_PKTS_EXP (41)
            Length: 8
        Field (3/7): TOTAL_FLOWS_EXP
            Type: TOTAL_FLOWS_EXP (42)
            Length: 8
        Field (4/7): FLOW_ACTIVE_TIMEOUT
            Type: FLOW_ACTIVE_TIMEOUT (36)
            Length: 2
        Field (5/7): FLOW_INACTIVE_TIMEOUT
            Type: FLOW_INACTIVE_TIMEOUT (37)
            Length: 2
        Field (6/7): SAMPLING_INTERVAL
            Type: SAMPLING_INTERVAL (34)
            Length: 4
        Field (7/7): SAMPLING_ALGORITHM
            Type: SAMPLING_ALGORITHM (35)
            Length: 1
    Padding: 0000

Below is the
copy of the Data Records which make use of the above Template but create no NFDump results as they have imcomplete data they appear to contain some sort of summary flow records (example is wireshark interpreation)

Frame 1: 102 bytes on wire (816 bits), 102 bytes captured (816 bits)
Ethernet II, Src: XX.XX.XX.XX Dst: XX.XX.XX.XX
Internet Protocol Version 4, Src: XX.XX.XX.XX Dst: XX.XX.XX.XX
User Datagram Protocol, Src Port: 4091, Dst Port: 9995
Cisco NetFlow/IPFIX
Version: 9
Count: 1
SysUptime: 3583899.052000000 seconds
Timestamp: Mar 13, 2020 14:12:29.000000000 AEDT
CurrentSecs: 1584069149
FlowSequence: 2
SourceId: 91
FlowSet 1 [id=256] (1 flows)
FlowSet Id: (Data) (256)
FlowSet Length: 40
[Template Frame: 19 (received after this frame)]
Flow 1
ScopeSystem: 0001
OctetsExp: 2499868
PacketsExp: 21858
FlowsExp: 3381
Flow active timeout: 60
Flow inactive timeout: 15
Sampling interval: 1
Sampling algorithm: Deterministic sampling (1)
Padding: 00

Note the above sampled netflow data records include no FLOW data just some sort of summary.
@phaag
Copy link
Owner

phaag commented Mar 28, 2020

Hmm .. difficult to tell from the what I see. Would it possible for you to send my a pcapd trace with all templates and some data records off list? You may use the email in the AUTHORS file.

@phaag
Copy link
Owner

phaag commented Apr 4, 2020

The exporter announces sampling with option elements IDs #34/#35.
This means that all data from this exporter are sampled likewise with
the announced sampling rate. There is no differentiation between
sampled/non sampled data, when using #34/#35.

If you have sampled and unsampled data from the same exporter, you need to
announce sampling with option elements #302/#304/#305 where element #302 - the
selectorID identifies the corresponding sampler per data record. Each data record
identifies the sampler by using tag #302

You can verify exporter and sampler any time later with the query:
./nfdump -E

@phaag
Copy link
Owner

phaag commented Apr 18, 2020

Any update on that otherwise I close the ticket

@ngner
Copy link
Author

ngner commented Apr 20, 2020 via email

@phaag
Copy link
Owner

phaag commented Jun 22, 2020

I close the issue. If you still need further invesigate this issue, please re-open it again.

@phaag phaag closed this as completed Jun 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@phaag @ngner