Spo unified #69

Closed
wants to merge 3 commits into
from

Conversation

Projects
None yet
2 participants

errzey commented Feb 6, 2013

This adds a new output plugin for writing to a single unified formatted log. This allows for multiple snort instances to run, each with their own by2 instance using this output plugin, and another single by2 instance for the final output.

Useful in situations where multiple instances of snort are running but you only want to maintain a single mysql connection open.

Bad things:
I do not have access to a non-unix'y operating system and I am sure that the file locking mechanisms are different on windows, so this will probably fail on win32. Could optionally ifdef it out for now.

Mark Ellzey added some commits Feb 6, 2013

Mark Ellzey Added output plugin for unified2 files.
In some cases a user may need to run multiple snort processes, each of them
outputting to different unified logs, though they want to maintain a single
barnyard instance for final logging (i.e., multiple snorts with a single
barnyard2 mysql connection).

This output plugin will allow multiple by instances to output to a single
unified2 file (file locking is included so no race conditions happen). This
allows for another by2 instance to read from this aggregated unified2 log and
output to some other plugin.

It should be noted that the generated files using epoch timestamp are rounded
down to the hour so that two instances of by2 will always write to the same
file.
73ce3d8
Mark Ellzey [spo_unified2] Added errors and cleanup functions 5174e99
Mark Ellzey [spo_unified2] fix resource leak. 9a17850
Collaborator

binf commented Feb 7, 2013

Im not really sure this should be part of the main code base.
Mabey you would like to advertise your feature on the ML and see what is the response.

There is a few kick back of using this setup especialy if people that do not know what their doing and use this output plugin to aggregate multiple segement into one and then do not understand what their seeing in their aggregated output.

I assume you are working with a PF_RING setup and you would want to Nth instance that monitor the same link to aggregate to a sub unified2 file?

I think a simpler solution would be to actually hook the spooler and add a configuration mode in the spooler that would directly write record to the aggregate file without transformation.

What is the reason why you want to limit to one db connection? Ressources?

You might want to start a thread on barnyard2-users/barnyard2-devel

Thanks.

errzey commented Feb 7, 2013

There is a few kick back of using this setup especialy if people that do not know what their doing and use this output plugin to aggregate multiple segement into one and then do not understand what their seeing in their aggregated output.

Not quite sure what this means. If a user doesn't know what something is, it's usually based on a lack of documentation or understanding.

I think a simpler solution would be to actually hook the spooler and add a configuration mode in the spooler that would directly write record to the aggregate file without transformation.

This would be easier, the callback system doesn't supply quite enough data to the plugins, so this means having to patch the spooler code itself. If this is the route you would suggest, design wise, a new hook type would be best. One that sits between the spooler and the output plugins, giving access to raw spooler structures.

What is the reason why you want to limit to one db connection? Ressources?

Yes.

Collaborator

binf commented Feb 7, 2013

On Wed, Feb 6, 2013 at 11:36 PM, Mark Ellzey notifications@github.com wrote:

There is a few kick back of using this setup especialy if people that do not know what their doing and use this output plugin to aggregate multiple
segement into one and then do not understand what their seeing in their aggregated output.

Not quite sure what this means. If a user doesn't know what something is, it's usually based on a lack of documentation or understanding.

Set it to understanding, if you monitor snort ML and by2 ML you will
see that some issue are recurent even if people use "guides" and this
is also valid most stuff out there.

But the point is that there is a difference betwen a deployment of
snort using PF_RING that spawn 8 process from the same interface and
having 8 snort process running on a system
each monitoring a different interface. In the later case you might not
want to aggregate unified2 data for multiples reason the first one
being that
if you are monitoring different interface, depending on your setup
they are probably monitoring different traffic and you could have
conflicting subnet that could generate alerts in different
context but have different meaning and this is why you would want to
have them alert independantly.

I think a simpler solution would be to actually hook the spooler and add a configuration mode in the spooler that would directly write record to the aggregate file without transformation.

This would be easier, the callback system doesn't supply quite enough data to the plugins, so this means having to patch the spooler code itself.
If this is the route you would suggest, design wise, a new hook type would be best. One that sits between the spooler and the output plugins, giving access to raw spooler structures.

The current spooler is legacy, but its not to hard to implement. In
the long run there will be a intermediate level betwen the input
plugin and the output plugin,
but it is not to hard to plug code at the spooler level that is
triggered by a global configuration flag.

If you want to check the new spooler code look @
https://github.com/binf/barnyard2/blob/dev-next (note that some stuff
will be updated soon especialy the event cache)
but you will get a updated idea on how its working.

What is the reason why you want to limit to one db connection? Ressources?

Yes.

If you log alot of data to a custom schema or even the default schema
but you do not need all the information, you can easely
slim down its memory requirement but tweaking the code a bit and it
should work like a charm (disabling cache, logging real
SID,GID,CLASS_ID etc..
So if you have 20 instances you can easely run by2 under 1m of ram,
it will just require you to slim some stuff.

errzey commented Feb 7, 2013

Thanks.

errzey closed this Feb 7, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment