Support type 152 field (fieldflowStartMilliseconds) and type 153 field(flowEndMilliseconds) #24

t-umeno · 2016-08-24T13:47:41Z

pmacct-1.5.2 uses template type 152 field (flowStartMilliseconds) and type 153 field(flowEndMilliseconds) instead of type 22 field (first_switched) and type 21 field (last_switched).

This patch supports:
150:flowStartSeconds
151:flowEndSeconds
152:flowStartMilliseconds
153:flowEndMilliseconds
154:flowStartMicroseconds
155:flowEndMicroseconds
156:flowStartNanoseconds
157:flowEndNanoseconds

This patch is based on:
https://github.com/logstash-plugins/logstash-codec-netflow/blob/master/lib/logstash/codecs/netflow/ipfix.yaml
https://github.com/logstash-plugins/logstash-codec-netflow/blob/master/lib/logstash/codecs/netflow.rb

flowStartSeconds flowEndSeconds flowStartMilliseconds flowEndMilliseconds flowStartMicroseconds flowEndMicroseconds flowStartNanoseconds flowEndNanoseconds Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

format_for_flowSeconds(time) format_for_flowMilliSeconds(time) format_for_flowMicroSeconds(time) format_for_flowNanoSeconds(time) Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

repeatedly · 2016-08-25T17:37:45Z

lib/fluent/plugin/parser_netflow.rb

-            event['last_switched']  = format_for_switched(msec_from_boot_to_time(event['last_switched'], pdu.uptime, time, 0)) if event['last_switched']
+          r.each_pair do |k, v|
+            case k.to_s
+            when /^(?:first|last)_switched$/


Could you check the performance to compare string and regexp?

string

case k when 'first_switched'.freeze when 'last_switched'.freeze when 'flowStartSeconds'freeze # ... end

regexp

case k when /^(?:first|last)_switched$/ when /^flow(?:Start|End)Seconds$/ # ... end

NetFlow-Generator genarates only NetFlow V5 packets.
https://github.com/mshindo/NetFlow-Generator

Please tell me how can I genarates NetFlow V9 test packets.
Otherwise I will make test program to check the performance to compare string and regexp.

t-umeno · 2016-09-01T05:43:16Z

$ ./benchmark3b.rb
0.460000 0.000000 0.460000 ( 0.539024)

#!/usr/bin/ruby
require 'benchmark'
print Benchmark.measure{ 1000000.times {
                           k = 'flowEndNanoseconds'
                           case k
                           when 'first_switched'.freeze then

                           when 'last_switched'.freeze then

                           when 'flowStartSeconds'.freeze then

                           when 'flowEndSeconds'.freeze then

                           when 'flowStartMilliseconds'.freeze then

                           when 'flowEndMilliseconds'.freeze then

                           when 'flowStartMicroseconds'.freeze then

                           when 'flowEndMicroseconds'.freeze then

                           when 'flowStartNanoseconds'.freeze then

                           when 'flowEndNanoseconds'.freeze then

                           else

                           end
                   }
}

t-umeno · 2016-09-01T05:45:58Z

$ ./benchmark4b.rb
1.360000 0.000000 1.360000 ( 1.551171)

#!/usr/bin/ruby
require 'benchmark'
print Benchmark.measure{ 1000000.times {
                           k = 'flowEndNanoseconds'
                           case k
                           when /^(?:first|last)_switched$/ then

                           when /^flow(?:Start|End)Seconds$/ then

                           when /^flow(?:Start|End)(Milli|Micro|Nano)seconds$/ then

                           else

                           end
                   }
}

t-umeno · 2016-09-01T05:47:48Z

So, I will rewrite with string.

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

repeatedly · 2016-09-05T06:29:56Z

lib/fluent/plugin/parser_netflow.rb

-            event['first_switched'] = format_for_switched(msec_from_boot_to_time(event['first_switched'], pdu.uptime, time, 0)) if event['first_switched']
-            event['last_switched']  = format_for_switched(msec_from_boot_to_time(event['last_switched'], pdu.uptime, time, 0)) if event['last_switched']
+          r.each_pair do |k, v|
+            case k.to_s


I'm not sure why original implementation calls to_s, but key is string so no need to_s call here.

I'm not sure why original implementation calls to_s, but key is string so no need to_s call here.

No!
case k.to_s is needed.
If you change case k,
'flowStartSeconds' doesn't match 'flowStartSeconds'.freeze.
Please check in your environment.

Hmm... it means 'k' is not string type?

'k' is Symbol type, so case k.to_s is needed.

r.each_pair do |k, v|
$log.warn "k.class: ", k.class
$log.warn "k.to_s.class: ", k.to_s.class
$log.warn "'first_switched'.freeze.class: ", 'first_switched'.freeze.class
case k.to_s

2016-09-06 22:02:03 +0900 [warn]: k.class: Symbol
2016-09-06 22:02:03 +0900 [warn]: k.to_s.class: String
2016-09-06 22:02:03 +0900 [warn]: 'first_switched'.freeze.class: String

Sorry for the late.
I understood the situation.
Better way is key = k.to_s before case and refer it in case and [].
It reduces the number of to_s call.

Thank you.
I rewrote along your advice.

t-umeno · 2016-10-07T01:24:33Z

master change: from k.to_s to key.
#53.3 failed
https://travis-ci.org/repeatedly/fluent-plugin-netflow/jobs/165544056

I think this failure is not caused by this change.
(issue of class Rational?)
Do you have any idea to resolve this error?

t-umeno · 2016-10-25T14:48:00Z

Are there any problems in this patch to merge ?

repeatedly · 2016-10-26T19:30:06Z

Why you revert key = k.to_s code?

repeatedly · 2016-10-26T20:12:50Z

Another idea is using symbol for case and use k.to_s in [].

repeatedly · 2016-10-26T20:21:02Z

case k
when :first_switched
  event[k.to_s] = ...
# ... other fields
end

Maybe, this is more faster than string case matching.

t-umeno · 2016-10-29T09:57:06Z

Thank you for you advice. I rewrite along this idea.

repeatedly · 2016-10-29T10:50:38Z

Thanks for the update.
I will check it later.

repeatedly · 2016-10-31T07:44:22Z

LGTM. Could you add a test?

t-umeno · 2016-11-03T04:04:53Z

Sorry, I can't add test codes immediately.

t-umeno · 2016-11-03T05:48:53Z

I'm reading this URL.
http://www.slideshare.net/repeatedly/dive-into-fluentd-plugin-v012

t-umeno · 2016-11-06T11:52:46Z

I add some test codes.
Please check these test codes.

repeatedly · 2016-11-07T09:50:48Z

Thanks!

t-umeno · 2016-11-07T14:48:40Z

Thank you!

t-umeno added 6 commits August 24, 2016 09:09

add: fields

b3c80c7

flowStartSeconds flowEndSeconds flowStartMilliseconds flowEndMilliseconds flowStartMicroseconds flowEndMicroseconds flowStartNanoseconds flowEndNanoseconds Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

remove: .to_i

70fbd85

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

add: function

def7a8c

format_for_flowSeconds(time) format_for_flowMilliSeconds(time) format_for_flowMicroSeconds(time) format_for_flowNanoSeconds(time) Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

remove: $log.warn 'flowset', flowset_id: flowset.flowset_id

a4d9478

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

repair: first_switched and last_switched

6a1c3c5

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

change: when /^(?:first|last)_switched$/

7bdb0fa

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

repeatedly reviewed Aug 25, 2016
View reviewed changes

change: rewrite with string

0d855cd

Signed-off-by: Takashi Umeno <umeno.takashi@gmail.com>

repeatedly reviewed Sep 5, 2016
View reviewed changes

change: from k.to_s to key.

fee137f

undo: use k.to_s

3f274e7

change: using symbol for case and use k.to_s in []

050179f

t-umeno added 3 commits November 6, 2016 18:23

Add tests for netflow v9 (flowStartMilliseconds)

2c31d49

Add tests for netflow v9 (flowStartMilliseconds)

f166773

Add tests for netflow v9 (flowStartMilliseconds)

7f0930f

repeatedly merged commit acb3c56 into repeatedly:master Nov 7, 2016

t-umeno deleted the flowStartMilliseconds branch November 7, 2016 14:48

Support type 152 field (fieldflowStartMilliseconds) and type 153 field(flowEndMilliseconds) #24

Support type 152 field (fieldflowStartMilliseconds) and type 153 field(flowEndMilliseconds) #24

Uh oh!

Conversation

t-umeno commented Aug 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

repeatedly Aug 25, 2016

Choose a reason for hiding this comment

Uh oh!

t-umeno Aug 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

t-umeno commented Sep 1, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

t-umeno commented Sep 1, 2016

Uh oh!

t-umeno commented Sep 1, 2016

Uh oh!

repeatedly Sep 5, 2016

Choose a reason for hiding this comment

Uh oh!

t-umeno Sep 5, 2016

Choose a reason for hiding this comment

Uh oh!

repeatedly Sep 5, 2016

Choose a reason for hiding this comment

Uh oh!

t-umeno Sep 6, 2016

Choose a reason for hiding this comment

Uh oh!

repeatedly Oct 5, 2016

Choose a reason for hiding this comment

Uh oh!

t-umeno Oct 6, 2016

Choose a reason for hiding this comment

Uh oh!

t-umeno commented Oct 7, 2016

Uh oh!

t-umeno commented Oct 25, 2016

Uh oh!

repeatedly commented Oct 26, 2016

Uh oh!

repeatedly commented Oct 26, 2016

Uh oh!

repeatedly commented Oct 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

t-umeno commented Oct 29, 2016

Uh oh!

repeatedly commented Oct 29, 2016

Uh oh!

repeatedly commented Oct 31, 2016

Uh oh!

t-umeno commented Nov 3, 2016

Uh oh!

t-umeno commented Nov 3, 2016

Uh oh!

t-umeno commented Nov 6, 2016

Uh oh!

repeatedly commented Nov 7, 2016

Uh oh!

t-umeno commented Nov 7, 2016

Uh oh!

Uh oh!

t-umeno commented Aug 24, 2016 •

edited

Loading

t-umeno Aug 26, 2016 •

edited

Loading

t-umeno commented Sep 1, 2016 •

edited

Loading

repeatedly commented Oct 26, 2016 •

edited

Loading