ELSA syslog-ng.conf rewrite r_pipes

[GoogleCodeExporter]

https://groups.google.com/d/topic/enterprise-log-search-and-archive/aSjnKuviZaQ/
discussion

rewrite r_pipes { subst('\|', "%7C", value("MESSAGE") flags(global)); };

Original issue reported on code.google.com by doug.bu...@gmail.com on 19 Dec 2013 at 11:20

[adepasquale]

Hello @dougburks, can I help with this somehow? Or is it on hold because of possible drawbacks?

[dougburks]

I'm working on new ELSA packages now. If you could help answer questions on our security-onion mailing list and/or test packages on our security-onion-testing mailing list, that will help expedite the process. Thanks!

[dougburks]

I have some preliminary packages in ppa:doug-burks/security-onion-dev that contain the latest ELSA code and this syslog-ng setting. It appears that the setting is throwing off some of our parsers, so there is more work to do. If you would like to test, please only install these packages on test machines.

[dougburks]

Only rewrite pipes for bro_* logs using the modified version here:
https://groups.google.com/d/topic/enterprise-log-search-and-archive/X6vExLLCT3g/discussion

rewrite r_from_pipes { subst('|', "%7C", value("MESSAGE") flags(global) condition(program("bro_*" type(glob)))); };

[dougburks]

New ELSA packages in ppa:doug-burks/security-onion-dev should be parsing correctly now. Please test on test installations.

[adepasquale]

I've tested package securityonion-elsa version 1205-1ubuntu0securityonion4 but I don't see the new rewrite rule:

# grep pipes /etc/syslog-ng/syslog-ng.conf 
rewrite r_pipes { subst("\t", "|", value("MESSAGE") flags(global)); };
        rewrite(r_pipes);

During the installation I got an error though:

Configurazione di sphinxsearch (2.1.9-release-0ubuntu15~precise)...
/etc/default/sphinxsearch exists, leaving alone.
sphinxsearch start/running, process 28678
Configurazione di securityonion-elsa (1205-1ubuntu0securityonion4)...
* Updating node mysql tables
ERROR 1125 (HY000) at line 1: Function 'archive' already exists
* Updating web mysql tables
* Updating web MySQL, please ignore any errors for this section...
* Finished updating MySQL
* Restarting ELSA web server.

[adepasquale]

My fault, I missed the securityonion-elsa-extras package...

Configurazione di securityonion-elsa-extras (20131117-1ubuntu0securityonion82)...
* Backing up /etc/syslog-ng/syslog-ng.conf to /etc/syslog-ng/syslog-ng.conf.20150505.
* Updating /etc/syslog-ng/syslog-ng.conf with rewrite r_from_pipes.
* Updating syslog-ng patterns.
* Restarting syslog-ng.
Error restarting syslog-ng.
* Backing up /etc/apache2/apache2.conf to /etc/apache2/apache2.conf.20150505.
* Setting Apache mpm_prefork_module MaxRequestsPerChild to 2
Enabling config file perl.conf.
To activate the new configuration, you need to run:
  service apache2 restart
grep: /home/*/.ssh/authorized_keys: No such file or directory
* Group exists. Checking Membership.
* /etc/elsa_web.conf has the correct group.
* /etc/elsa_node.conf has the correct group.
* Backing up /etc/elsa_web.conf to /etc/elsa_web.conf.20150505.
* Restarting ELSA web server.

The syslog-ng restart error was due to a custom destination which I removed, and then syslog-ng restarted just fine. Going to try some ELSA web searches now.

[adepasquale]

I can confirm that now I see BRO_HTTP logs with %7C inside the message, and all fields are parsed correctly. 👍

[dougburks]

Thanks for testing!

[dougburks]

Submitted for testing:
https://groups.google.com/d/topic/security-onion-testing/OHhNEapIUgE/discussion

[dougburks]

Published:
http://blog.securityonion.net/2015/06/elsa-1205-now-available.html