new alert_json plugin with kafka capabilities #88

wants to merge 132 commits into


None yet

5 participants


Hi, we have created a new alert_json plugin that provides to barnyard2 the posibility of send alert in json format to a file or to kafka messaging system. The idea is to provide bigdata support to barnyard.

Please, review README and if you need more detail, contact us.

root and others added some commits Mar 28, 2013
root Created alert_json output plugin skeleton, and integrated in banyard2.
     modified:   src/output-plugins/
     modified:   src/plugbase.c
     new file:   src/output-plugins/spo_alert_json.c
     new file:   src/output-plugins/spo_alert_json.h
root Changed names under spo_alert_json. No funtionality changed except de…
…fault output names

      modified:   output-plugins/spo_alert_json.c
root Timestamp now printed in milisenconds, instead of string eb71ecc
root Each data type add it's own string to a json file now. ec29434
root FIXED: JSON invalid fields are not written at all. Before, a ",," was…
… printed

      modified:   output-plugins/spo_alert_json.c
root added output-plugins/spo_alert_json.h in src/plugbase.c
	modified:   src/plugbase.c
root Added kafka libraries and header (in a future we will add the entire
	new file:   output-plugins/kafka/librdkafka.a
	new file:   output-plugins/kafka/rdkafka.h
	new file:   output-plugins/librdkafka.a
root Added kafka output to spo_alert_json. Topic is need to know in compile
time, next commit will fix that.
Created sfutil/sf_kafka, to send kafka messages. Modify some
and added -lz, -lrt and -lpthread c flags, needed by kafka. Added
rdkafka library too.

	modified:   output-plugins/spo_alert_json.c
	modified:   sfutil/
	new file:   sfutil/kafka/librdkafka.a
	new file:   sfutil/kafka/rd.h
	new file:   sfutil/kafka/rdaddr.h
	new file:   sfutil/kafka/rdcrc32.h
	new file:   sfutil/kafka/rdfile.h
	new file:   sfutil/kafka/rdgz.h
	new file:   sfutil/kafka/rdkafka.h
	new file:   sfutil/kafka/rdrand.h
	new file:   sfutil/kafka/rdtime.h
	new file:   sfutil/kafka/rdtypes.h
	new file:   sfutil/sf_kafka.c
	new file:   sfutil/sf_kafka.h
root Kafka topic can be specified adding a '@' after broker's name
	modified:   output-plugins/spo_alert_json.c
root FIX: alert_json can send alerts to a file and a kafka broker at the same
time now
	modified:   output-plugins/spo_alert_json.c
root Delayed KafkaLog's handler init in daemon mode (Need to do because a
fork() in that mode).
	modified:   src/output-plugins/spo_alert_json.c
	modified:   src/sfutil/sf_kafka.c
	modified:   src/sfutil/sf_kafka.h
root Increased spo_alert_json LOG_BUFFER; Kafka split messages with just 4K
	modified:   src/output-plugins/spo_alert_json.c
root Changed the way sf_kafka use the buffer. Now it allocate a new one an…
…d let

librdkafka free it.
	deleted:    src/output-plugins/kafka/librdkafka.a
	deleted:    src/output-plugins/kafka/rdkafka.h
	modified:   src/sfutil/sf_kafka.c
	modified:   src/sfutil/sf_kafka.h
root FIX: When sending alerts, sometimes proto was not set, so the json alert
contained a blank space in arguments, surrounded by commas (", ,"). Now,
if the alert's proto is not valid, we don't send the comma.
	modified:   src/output-plugins/spo_alert_json.c
@binf @eugpermar binf Bumped: version to 2-1.13-BETA
Bumped: build to 325

Add: Full support for sid-msg v2 format which
     enchanced by the following fields: gid,revision,classification,priority
     for each entry which allow pre-population of signature metadata by
     barnyard2 if database output is used.

Add: Signature Suppression support at the spooler level using
     configuration directive. See doc/README.sig_suppress

Add: Variable resolving/support in configuration file
     (generic variable.

Add: hostname and interface to possible CSV field
     Feature requested by: Phil Daws

Add: spo_database configuration keyword "disable_signature_reference_table"
     was added and reconnect_sleep_time, connection_limit defined in

Fixed: Added extra check when generating sig_reference cache.
       (Martin Olsson)

Fixed: and double declaration issue (using
       command line and directive is now prohibited) [ will bail
       if both are used (-S and config sid_file OR -G and config

Fixed: syslog_full in complete mode IP information (F�bu Hufi)

Fixed: database, could stop processing event when some ip options where
       null (John Naggets)

Fixed: Removed some database messages and move them to debug message if
       the propre debug flag is used.
@binf @eugpermar binf Last minute commit for a long waited needed feature and some little fix.
Add: Support for proper signal handling.
Add: README info for google mailing lists.

Fixed: Compile issue when debug was enabled (missing , in some

Fixed: Changed a few places where the snort literal was used instead of
barnyard2 and this could confuse some first time barnyard2 users.

Fixed: RPM spec file to point to good version (when needed)

Bumped: Build to 326

--github specific
Fixes #81
Fixes #73
Fixes #75

Close #82
Close #83
Close #80
Close #79
Close #78
Close #27
--github specific
@firnsy @eugpermar fixed: libwebsocket update collapsed a number of arguments into gener…
…al struct.
@firnsy @eugpermar updated: build bump and removal of beta tag pending release. bcbd6b0
@firnsy @eugpermar fixed: range logic was inadvertenly inverted. 33f49f8
@firnsy @eugpermar fixed: lingering reference identifid during HUP operations. b0ccd22
@firnsy @eugpermar added: handle situations where map files are not v2. 317f4be
@firnsy @eugpermar fixed: issue with signature insertion and v1/v2 handling. d856c41
@eugpermar root Parsed hosts and networks file in order to produce in spo_alert_json 5fefbe0
@eugpermar root Added src_name, src_str, dst_name and dst_str to json fields, based on
/opt/rb/etc/{hosts,networks} files
	modified:   output-plugins/spo_alert_json.c
@eugpermar root Added network identifications in source and destination ip
	modified:   output-plugins/spo_alert_json.c
@eugpermar root Added geoIP support using maxmind GeoIP
	modified:   output-plugins/spo_alert_json.c
@eugpermar root Fixed a possible memory leak.
	modified:   src/sfutil/sf_kafka.c
@eugpermar root Fixed some possible memory leaks and enabled --enable-geo-ip in
configure script
	modified:   src/output-plugins/spo_alert_json.c
@eugpermar root Updated librdkafka 974bef5
@eugpermar root Added to sf_kafka a maximum queue length value
	modified:   sfutil/sf_kafka.c
@eugpermar root Added json_output plugin 7f1de11
@firnsy @eugpermar fix: possible double free's on cleanup when HUP recieved. 563bc89
@eugpermar @eugpermar eugpermar Added json and geoIP libs conditional compile stuff in
KafkaLog now have a TextLog to print to a file, and alert_json does not
have to worry about send to kafka and to a textfile.
@eugpermar eugpermar FIX: sf_kafka did not compile if --enable-kafka were passed to compile
@eugpermar eugpermar Added priority and classification fields to alert_json plugin. Some f…
…ields will be always printed, too
@eugpermar eugpermar Added sensor_id capability f4cb1c3
@eugpermar eugpermar Added sensor_id and sensor_name passed by params. We will send the co…
…untry code in s_ip and dest_ip too.
@eugpermar eugpermar Using sf_ip from sfutils module, what simplifies a bit the name resol…

Managed a geoip NULL return when ip is not in the database.
@eugpermar eugpermar FIXED: Sendig src_country twice by error b03bb29
@eugpermar eugpermar Added spread over kafka partitions capacity. 2b98f85
@eugpermar eugpermar Deleted kafka libs and used system ones dfea910
@eugpermar eugpermar The geoip library path is now passed by param. 67a8c03
@eugpermar eugpermar Readed services and protocols file, and sended in the event 44ad609
@eugpermar eugpermar FIX: dst port name was the same as src_port name. 5a8741d
@eugpermar eugpermar FIX: Classification field now show the correct classification message 05fcdc2
@eugpermar eugpermar Created rbutil folder, where it will be all redborder utils. aacf0d9
@eugpermar eugpermar Created rbutil folder, where it will be all redborder utils. 7e2eda9
@eugpermar eugpermar Amend: fixed src/ cb52266
@eugpermar eugpermar Warning supressed. dba7696
@eugpermar eugpermar Changed -DJSON_KAFKA and -DJSON_GEOIP to defines in config.h file 8ac1090
@eugpermar eugpermar Added payload field 3f2a03a
@eugpermar eugpermar Changed the strcmp alerts processing system to a template based one 99adef9
@eugpermar eugpermar Using the json name given in the template 9a808be
@eugpermar eugpermar Added a "default value" field in the template 3e97959
@eugpermar eugpermar Update doc 9758863
@eugpermar eugpermar Proto fallback value is now the same value as proto_id 38f3cac
@eugpermar eugpermar Added ARP and VLAN parsing. 262c850
@eugpermar eugpermar Resolved VLAN names. f8519e9
@eugpermar eugpermar FIX: ipv4 were not passed by ntohl functions 8fd7109
@eugpermar eugpermar Deleted all slow KafkaLog_Print and changed to KafkaLog_Puts. Added a…
… _itoa function to convert number to string. Template default values changed from void* to char*
@eugpermar eugpermar changed a snprintf to a strcat in KafkaLog_Write: performance
KafkaLog->maxBuf renamed to kafkaLog->bufLen and start_bufLen, more descriptive ones
@eugpermar eugpermar Changed hosts format to [name addr] (See redmine #522 issue) c7477a5
@eugpermar eugpermar Timestamp resolution moved from miliseconds to seconds 529e965
@eugpermar eugpermar alert-json did not work if --enable-ipv6 was present (Redmine issue 6…
…20). Fixed
@eugpermar eugpermar Changed so it can import rdkafka from any location. Now you
can specify kafka location using --with-kafka-* params configure params.
@eugpermar eugpermar Added rb_pointers.h header to have specific commands to check pointer…
…s. Added type,domain,domain_id template parameters too.
@eugpermar eugpermar Geoip libraries can now be specified in configure (--with...) f7eff47
@eugpermar eugpermar Sending action of message (not fully supported). Some numbers sended …
…in hex format => json not supported.
@eugpermar eugpermar IPv4 sended in wrong format. Deleted a warning. fwsam did not compile…
… if ipv6 enabled.
@eugpermar eugpermar Barnyard could wait forever if the kafka broker was down. Fixed. 2da769a
@eugpermar eugpermar Eth address not printed with all zeroes padding. Fixed. 038d63a
@eugpermar eugpermar Added kafka 0.8 support 55859fc
@eugpermar eugpermar Workaround to solve bug in librdkafka. 003086d
@eugpermar eugpermar Changed some fields name. e11344d
@eugpermar eugpermar Deleted sensor_id_snort from default template 0665f85
@eugpermar eugpermar Changed some template elements 9bc12e3
@eugpermar eugpermar Changed some template names a3a04f1
@eugpermar eugpermar Deleted kafka 0.8 support. Fixed memory leak. Inserted some likely()
@eugpermar eugpermar Updated 4d4b105
@eugpermar eugpermar Added group_id and group_name in the parameters. domain_id renamed to…
… domain_name, and domain->domain_id
@eugpermar eugpermar IP packets length and ethernet packets length are now aggregated in g…
…roups. Priority_name added.
@eugpermar eugpermar Added AS numbers 8f6035a
@eugpermar eugpermar Changed Fatal errors to error messages. 0f2e165
@eugpermar eugpermar Update to current kafka api 770955c
@eugpermar eugpermar FIX sometimes print a *0 when printing ethlength. d24420d
@eugpermar eugpermar FIX: ntohl were not applied in ip. 9bcd10b
@eugpermar eugpermar Solved a invalid write valgrind report. Don't sending ip,mac,vlans na…
…mes by default.
@eugpermar eugpermar Added rb_macs_vendor support ab3c3da
@eugpermar eugpermar Cleaned old kafka code 5086948
@eugpermar eugpermar FIX: ETHDST_VENDOR without mac in switch. 4b33b2f
@eugpermar eugpermar FIX: spo_alert_json didn't compile if not HAVE_GEOIP present 4fa91db
@eugpermar eugpermar FIX: rb_kafka didn't compile if not --enable-rdkafka present (thanks …
…to Alberto for reporting)
@eugpermar eugpermar FIX: ethlen puts an extra 0 sometimes 56c8d63
@eugpermar eugpermar You can pass option to rdkafka directly (See #2250) f272e43
@eugpermar eugpermar Added delivery function callback for every message ac57192
@eugpermar eugpermar Deleted priority name. Now priority is always a name 34376d1
@eugpermar eugpermar FIX: Buffer overflow e3baa9b
@eugpermar eugpermar FIX: Buffer overflow 37a8067
@eugpermar eugpermar FIX: priority was sent even when there was no event 3ab793e
@eugpermar eugpermar FIX: Bad delivery message callback management d653674
@eugpermar eugpermar Barnyard cache is now freeing at the end, instead of start. This avoi…
…ds a lot of lonely packet events
@eugpermar eugpermar Sanitized spooler.c: some functions made static, some prototypes dele…
@eugpermar eugpermar Enabled lonely events processing. SRC/DST IP are now extracted from e…
…vent first.
@eugpermar eugpermar Trying to get proto from event before attempt to extract from the pac…
@eugpermar eugpermar src and dst port now extracted from the event instead of the packet. …
…If cannot extract from event, then try to extract from packet.
@eugpermar eugpermar ICMP type now tried to extracted from event first than the packet 0f49c62
@eugpermar eugpermar extract icmp code first from event that from packet. ICMP code & type…
… only printed if icmp protocolot event
@eugpermar eugpermar Trying to extract vlanId information of the event first of the packet. f00d29d
@eugpermar eugpermar ipv6 country database loaded separately from ipv4 country database e103aac
@eugpermar eugpermar Autonomous System Numbers ipv6 database are now sepparated from ipv4 …
@eugpermar eugpermar Cosmetic changes: SRC_REQ and DST_REQ uses are now encapsulated by ma…
@eugpermar eugpermar FIX: didn't compile if --enable-ipv6 was not present 03555b0
@eugpermar eugpermar rb_pointers added copyright 7601011
@eugpermar eugpermar Extracted actionOfEvent function, in order to reuse in another output…
… plugin
@eugpermar eugpermar Syslog output plugin can print the action taken 5fa044b
@eugpermar eugpermar Syslog output plugin now prints action taken too. 9f5c7ad
@eugpermar eugpermar Added sensor name to output 078102a
@eugpermar eugpermar Added sensor-group to syslog output plugin 485110f
@eugpermar eugpermar Renamed "sensor name"->"sensor" and "sensor group"->"group" 7099fe5
@eugpermar eugpermar Added rbutils/, and headers guards b46067c
@eugpermar eugpermar Increased syslog human-readable IP buffers size, in order to accomoda…
…te ipv6
@eugpermar eugpermar FIX: Ports were using as uint8_t, instead of uint16_t a38c808
@eugpermar eugpermar FIX: bad IPlength 6099b36
@firnsy firnsy added the feature label Nov 5, 2014
maxtors commented Sep 16, 2015

This PR seems very interesting! What is the status of a potential merge?

Hi @Maxtors !

Currently, we are using this kafka plugin at redBorder, but we have not introduced the last changes made by firnsy in this repo (but we have the intention to do it). This has caused some small conflicts between both, so they can't not be merged automatically at this moment.

You can see them in:

As @jjptapia said, don't hesitate to contact us if you need something related with this output plugin!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment