IPFIX (Netflow v10) support #10
Conversation
|
I'm anxious to use this because I want to get Netflow form our ASAs into Elasticsearch. Any idea when this will be ready to merged? Or is there an easy way to try it out in the mean time? I'm not sure if I need to actually build a new gem or if I can just clone this into the existing plugin directory inside the Logstash install. |
|
I think it's good enough to go, I've tested it against both However if you're really after Netflow (and I'm not 100% sure if ASA's support IPFIX) then this PR won't be much use; the good news is the plugin already supports Netflow v9 although there are probably missing field definitions for some of the more esoteric stuff that I think an ASA can export. The README details how to run this plugin within a Logstash install. |
|
Hi, I try to use this plugin with a IPFIX probe. My config is: Have you any ideas ? |
|
I finally managed to get this installed properly (sorry it took me so long). I think the ASA is presenting a rather difficult situation: it is sending Netflow v9 data but it also contains ipfix fields. When I set the version to 10, logstash ignores the data: When I set the version to These definitions do exist in the The most confusing part is even when I create a custom I realize this probably has nothing to do with your additions to the plugin, but any help you can offer is greatly appreciated since you seem to understand how this plugin works more than I do. |
|
Like @minou, I've been getting consistent `SystemStackError: stack level too deep' within a couple of minutes of starting logstash. I've been testing with the following Netflow probes, both with Netflow v10 options.
Running Logstash-1.5.3 from RPM with a git clone from your ipfix branch with this config: Errors: It also dumps .inspect style stuff on stdout, seemingly for every flowset, but I don't see where it is coming from: |
|
The tracing is from the BinData gem, just remove the |
|
I'm not sure how to go about debugging the stack level issue, I've seen it only a couple of times but then on a subsequent run it doesn't occur at all so I'm at a loss as to what to do there. |
|
@samdoran The ASA is only sending Netflow v9. The error you're getting is not from missing field ID's, but because the ASA hasn't sent a template packet to Logstash. This is often configurable on the device, either to send templates every N packets or ever X minutes/seconds. If you still get unknown fields then these are not necessarily the same as the equivalent IPFIX field in PEN 0. For ASA I would refer to the Cisco documentation, a quick search turned up this: Some of these fields are "standard" for Netflow v9, some are specific to the ASA. |
|
Ah it runs better with the BinData::trace_reading commented out. |
|
Maybe it is literally just having tracing enabled. I'll push a change to remove it. |
|
I've been happily running @bodgit ipfix branch for almost 2 months now with 5 netflow exporters (softflowd and ipt_netflow both exporting ipfix). |
|
Are you still using this for IPFIX @jorritfolmer? I’m accepting v10 inputs using this codec and getting a similar error. |
|
Yes, I'm using the latest IPFIX branch from @bodgit, including the "Remove bindata tracing" commit. |
|
@jorritfolmer I have it setup and are able to take NetFlow V9 etc but once I try send any IPFIX data I get the below: {:timestamp=>"2015-10-28T11:22:07.045000+0000", :message=>"Unsupported enterprise", :enterprise=>6876, :level=>:warn} Can't see any info regarding this anywhere (scratches head) |
|
Ah cool! It appears you have a VMware dvSwitch exporting IPFIX? |
|
Sure @jorritfolmer will do. Thanks. |
|
Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'. |
|
Am anxious to test this. Has it been committed? |
|
Mergeeeeee pleeeeeeease! |
|
Quick question, is there any plans to support in the future Enterprise-specific Information Elements transformations. In our organization we would like to transform Sonicwall IPFIX (PEN 8741) flows to another proprietary "standard" from Fluke Networks. |
|
Given reports of success of this branch, I'm OK merging it. |
|
This PR fails to patch into master for me: |
|
|
Sorry, I didn't see these email notifications, I'll try and rebase again shortly. |
|
i've learned an awful lot about git today. to get this patch to work before an update: |
|
@nobletrout just for clarification, you're using the patch successfully with IPFIX? |
|
I have been using this patch now for a two weeks and it works fine. Also I have changed the ipfix.yaml file in order to accommodate Sonicwall IPFIX format. |
|
Need support for variable length fields. Looks like the choice of using On Thu, Feb 11, 2016 at 2:48 PM, RobertLukan notifications@github.com
|
|
I do not have issue for the flows from Sonicwall, data has fixed length. For sure it would be good to have variable length field. |
Rename some things to make them Netflow-specific.
- Add a basic BinData PDU record that just treats all non-header data as a flat string - Generate an equivalent YAML file for IPFIX field definitions. IANA provides a CSV with the field definitions so created a simple script to parse that and then generate the reverse field definitions as per RFC 5103
Currently the code cowardly refuses to deal with variable length encoded fields or any of the complex data types, skipping any template containing them. Tested with `softflowd`. Currently no field is "fixed up" such as timestamps; IPFIX unlike Netflow does not include the uptime of the exporting device in each packet. However testing with `softflowd` shows an initial option record is sent with the systemInitTimeMilliseconds field which is what is needed for subsequent calculations so maybe that should be cached per-device while packets are received with some frequency. A device however may also include this field as part of its normal templates.
Just copy the Netflow v9 tests and generate a small IPFIX flowset.
Testing with OpenBSD's `pflow(4)` device a) worked and b) highlighted that it
exports using the flow{Start,End}Milliseconds fields which contain an absolute
timestamp. Fix this up to be a regular ISO 8601 timestamp and also add support
for the equivalent seconds, microseconds, and nanoseconds fields.
Possibly causes stack errors having it enabled?
Based on similar changes to Netflow v9 code.
|
That should now be rebased against master |
|
I really appreciate the work that has been put into this. But I don't think it's ready to be merged. I'm exporting ipfix from netscalers, and the "Cowardly" variable field lenght, means my data-templates isn't imported. That means all the interesting data is thrown away. :( If I can contribute to this plugin in any way, let me know. |
|
It's not necessarily the variable length fields that are the problem but they're commonly used with the IPFIX complex data types; basicList, subTemplateList & subTemplateMultiList and these are difficult to implement as they have the potential to be nested within each other so it adds an element of recursion to parsing the packets which isn't present with Netflow <= 9. I'm not even sure how you'd visualise or index a basicList of subTemplateMultiList's in Elasticsearch for example. I initially tested with Can you possibly include a visualisation/export of your netscaler data templates captured with something like wireshark so we can see just how complicated they are? |
|
Thank you for taking the time to reply. I've already debugged it with wireshark. I can see the templates coming in: http://i.imgur.com/tCpCA2y.png When the templates come in, i see the warning you put in the netflow.rb ie "Cowardly refusing to deal with variable length encoded field" Some of the non-variable length data templates is working, and i do get data into my elk: http://i.imgur.com/WetIJtd.png I'm exporting ipfix from a netscaler. The netscaler exports with a PEN of 5951, which i've added to ipfix.yaml. But since the templates aren't accepted, i get no data. :( Let me know if you need pcaps or anything else. |
|
Hello, I am trying to get IPFIX flows into ELK for monitoring and ran across this plugin. Since IPFIX is not yet supported in the logstash's netflow codec, I figured I would give this a try (for development of course). I'm somewhat new to working with logstash, so would anyone be able to give me some tips on the best way to install this plugin? In case it matters, I am trying to bring in IPFIX flows from a Juniper MX. Thanks for any help. |
|
I went ahead and rebased this PR in bodgit-ipfix branch, with conflicts resolved.
Objections to merging bodgit-ipfix branch into master? |
|
👍 it would be good to get this finally closed. |
|
Merged! |
|
RobertLukan... Can you share with me your changes to ipfix.yaml for Sonicwall? |
I've been meaning to do this for a while however every time I got waylaid the layout of the Logstash source changed and I had to try and rebase my changes. Now the codec is split into its own repository I should be able to get this done :-)
IPFIX is very similar to Netflow v9 in design however there are some subtle differences:
This PR is not yet complete so please don't merge yet but I thought I would open it here so people could see it's progress and/or comment.