Elastic Stack Alpha Release

[dougburks]

• Elasticsearch

  • Docker container
  • run as a non-root user
  • log to /var/log/elasticsearch/
  • add script to reset Elasticsearch database
  • --env "bootstrap_memory_lock=true"
  • --ulimit memlock=-1:-1
  • --env ES_JAVA_OPTS="-Xms$ELASTICSEARCH_HEAP -Xmx$ELASTICSEARCH_HEAP"
  • set initial $ELASTICSEARCH_HEAP based on available RAM
  • MAX_OPEN_FILES=65536
  • elasticsearch - nofile 65536
  • elasticsearch soft/hard nproc 2048
  • field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.

• Kibana

  • Docker container
  • run as a non-root user
  • log to /var/log/kibana/
  • set source_ip and destination_ip fields to type "ip"
  • dashboard for each Bro log
  • resolve field type conflict for status_code field
  • remove ELSA shortcut from desktop and replace with Kibana
  • Top Source/Destination by Total Bytes
  • add source port to NIDS dashboard
  • configure Kibana index for 1 shard and 0 replicas
  • Indicator dashboard should only search last 24 hours by default
  • Tables should show at least 10 results (perhaps 100)
  • add visualizations and dashboard for Bro syslog.log
  • add visualizations and dashboard for standard syslog coming from syslog-ng
  • DNS dashboard needs to be adjusted for recent navigation menu change (scroll bar)
  • comment out unnecessary config in /etc/kibana/kibana.yml
  • build new Kibana image with links to Squert and Logout in the side bar
  • add message visualization to Notices dashboard
  • add sub_message visualization to Notices dashboard
  • DNS query field should hyperlink to Indicator dashboard
  • SSL Server Name field should hyperlink to Indicator dashboard
  • remove Squert and Logout links from Navigation visualization in favor of new links in side bar
  • HTTP site (virtual_host) field should hyperlink to Indicator dashboard
  • Intel indicator field should hyperlink to Indicator dashboard
  • Bro Files - Source IP field should hyperlink to Indicator dashboard
  • move ElastAlert menu entry from Other to Alert Data
  • add dashboard for ElastAlert
  • fix index-pattern-field(s) for Sysmon
  • fix index-pattern-field(s) for Autoruns
  • source_port field should hyperlink to Indicator dashboard
  • destination_port field should hyperlink to Indicator dashboard
  • set timepicker:timeDefaults to search last 24 hours
  • set dashboard:defaultDarkTheme to true
  • Bro Connections maps are showing a scroll bar for the navigation menu, needs resizing
  • Firewall dashboard has multiple "could not locate that index-pattern-field" errors

• Logstash

  • Docker container
  • run as a non-root user
  • log to /var/log/logstash/
  • fix typo in filename 0001_intput_json.conf
  • add filters for bro_modbus, bro_pe, bro_rfb, bro_sip, bro_radius, bro_dnp3
  • change event to event.set and event.get where necessary
  • only add bro tag if type is bro
  • move rename from 0000_input to 1000_preprocess
  • comment out 0003_input_syslog.conf
  • update Bro patterns for Bro 2.5 (securityonion-elsa-extras: update for Bro 2.5 #1029)
  • add parser for OSSEC alerts
  • add parsers for Bro SMB
  • parse Bro Notice fields message and sub_message as msg and sub_msg
  • retain IPv6 addresses
  • add logstash_time field
  • persistent queues
  • set pipeline.workers: 1
  • NIDS alert - lookup rule in /etc/nsm/rules/ and insert into alert
  • parse pfSense firewall logs
  • parse Sysmon logs
  • parse Autoruns logs
  • rebuild logstash image with new logstash-filter-rest plugin
  • update pfSense parser with fix for grokparsefailure on ICMPv6 traffic
  • update Snort parser to parse sensor name

• Curator

  • Docker container
  • run as a non-root user
  • log to /var/log/curator/

• ElastAlert

  • Docker container
  • run as a non-root user
  • log to /var/log/elastalert/
  • disable elasticsearch replicas for elastalert_status and reduce shards to 1
  • default rules should use use_count_query

• Freqserver

  • Docker container
  • run as a non-root user

• Domainstats

  • Docker container
  • run as a non-root user

• so-elastic-configure

  • check for options in /etc/nsm/securityonion.conf before adding them

• so-elastic-configure-elastalert

  • check to see if elastalert_status index exists before deleting

• so-elastic-start

  • --link is a deprecated legacy feature, replace with user-defined network
  • need a timeout value when checking for Elasticsearch so that script doesn't hang forever
  • provide prettier/quieter message when container(s) already started

• so-start / so-stop / so-restart

  • create small wrapper scripts that control both NSM and Elastic
  • create /usr/sbin/so-status

• sostat

  • provide stats for Elastic stack

• soup

  • when docker is upgraded, it stops all containers, need to restart

• ELSA

  • dump ELSA data out to filesystem in a form that Logstash can read
  • have Logstash read the original Bro timestamp and use that for ingestion into ES

• CapMe

  • move to single sign-on
  • update for cross cluster search

• Sguil

  • change ELSA pivot to Kibana search for that IP on Indicator dashboard

• Squert

  • pivot from IP address to Kibana search for that IP on Indicator dashboard
  • move to single sign-on

• Setup

  • add Elastic option to Setup for new installations but mark it Experimental
  • exclude docker interfaces from interface selection

• OSSEC

  • rootcheck is alerting on Files hidden inside directory '/var/lib/docker/aufs
  • opening multiple kibana tabs may result in active response "High amount of POST requests in a small period of time (likely bot)", check for /es_admin

• distributed deployments

  • Cross Cluster Search

• update existing packages

  • securityonion-setup
  • securityonion-sostat

• build new packages where necessary

  • securityonion-elastic
  • securityonion-samples-bro

• develop tests for QA

  • check for _grokparsefailure, _csvparsefailure, _rubyexception
  • check log files:
    • /var/log/elasticsearch/
    • /var/log/logstash/
    • /var/log/kibana/
    • /var/log/elastalert/
    • /var/log/curator/
    • /var/log/freqserver/
    • /var/log/apache2/*

[JonZeolla]

Not sure if it's helpful, but not too long back I put together a wiki article covering how to tune ES, focused on bro data for another project. It is intended for 2.4, and I noticed you recently upgraded to 5.4, so I'm not sure if this is still valid.

[dougburks]

Thanks @JonZeolla !

[ragdelaed]

are there any plans for a hadoop/big data enabled back end for SO?
or do you plan to use kafka, storm, zeppelin or any other tools from the hadoop ecosystem?
also any profiling, baseling, or anything like that for statistical or machine learning analysis?
and do you plan to pull in windows event logs in the future in addition to bro/snort/etc?

thanks for your work on this, its an excellent product, elegant, and it fits in my brain nicely. :)

[dougburks]

Hi @ragdelaed ,

You can pull in Windows Event Logs today using OSSEC agent or several other event log collectors.

Right now, our main focus is integrating the Elastic stack. Once that is done, we'll look at what makes the most sense for our next project.

Thanks!

[dizzykc]

@dougburks - This migration is AWESOME. I have already taken version one if your script and turned it into a fully functioning AWS AMI. As a proof of concept 1:1 ratio (management to sniffing interface), I have collected data to show my teams what type of traffic is coming into our system. I have begun to play around with ElastAlert, but running into some index pattern issues. I have showed this off to my teams enough times that once the migration to ELK is completed, we are going to be implementing this across the entire company. All I can say is JOB WELL DONE SIR and THANK YOU!

FYI - I am hoping to be at the Security Onion conference this year. Very excited!

[dougburks]

Hi @mv003348 ,

Thanks for the feedback! Hope to see you at the conference!

[gaganova]

Is this transition to elastic stack something that will definitely happen? Why? I would be very interested in reading about your reasons for this decision. Don't you think ES is too much bloat?

[dougburks]

Hi @gaganova ,

This is most likely going to happen. Our community and our customers have been asking for the Elastic Stack for quite some time and we try to be responsive to demand.

[gaganova]

I am asking myself if people demand that because they have only seen nice ELK screenshots or if they have actually used that bloated java stack in real life. Of course there is no answer to that, so hopefully this project will survive this decision. Good Luck! Anybody interested in maintaining the ELSA based version with 16.04?

[dougburks]

Hi @gaganova ,

Replies inline.

I am asking myself if people demand that because they have only seen nice ELK screenshots or if they have actually used that bloated java stack in real life. Of course there is no answer to that, so hopefully this project will survive this decision. Good Luck!

I can tell you that there are many folks in our community who have already rolled their own Elastic deployment and are quite happy with it.

Anybody interested in maintaining the ELSA based version with 16.04?

While ELSA certainly has low resource requirements, that results in quite a few drawbacks which limit hunting capabilities (only 12 indexed fields, no multiple groupby, just to name a few). These limitations caused the lead developer of ELSA to then migrate to Elasticsearch for what was going to be ELSA 2.0. The future of ELSA 2.0 is currently unclear, thus we are moving to the Elastic stack.

[mattraneri]

Hi @dougburks,

First and foremost, love this project. I would love to add the ELK stack to my deployment, but is there a timeline of when it will officially be supported? I'd hate to implement this in our deployment and have the official project roll out ELK two weeks later. Just asking because the last blog post was all I was able to find directly referencing the script, which is now months old.

Thank you!

[dougburks]

Hi @mattraneri ,

We've had a few blog posts about the Elastic integration:
http://blog.securityonion.net/search/label/elastic

We're not ready to commit to any timelines at this point. This is a large project and we still have a lot of work left to do. This is an important project and we want to make sure that it's done right!

Thanks!

[dizzykc]

@dougburks - I have been following the updates and launching new EC2 instances every time a new TP comes out. All I can say is, :mind blown:. This stuff is AWESOME! Whenever it becomes a stable version, I am making it our Enterprise IDS tool...all I need now are boat loads of SO stickers! THANK YOU!

[ghost]

Hi @dougburks,

Congratulations, this integration is awesome.

Where are the dockerfiles o how manage the containers? How is the deployment ?

THANKS !!!

[ghost]

Sorry I've already found docker images.

But I try config with Production Mode and I have problems with dockers.

I would like test the integration to do an academic work,
maybe, Could I open other issues?

[weslambert]

@kxuanCeltik ,

Please see: http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html

This technology preview only works with Evaluation Mode currently.

Instead of opening an issue, or commenting on this issue, please use our mailing list:

https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists#security-onionv

Also see: https://groups.google.com/forum/#!topic/security-onion/Sr8wrJSQxo0

Thanks,
Wes

[ghost]

Ok, thanks.

[behkxyz]

@dougburks just to add some notes https://groups.google.com/forum/#!topic/security-onion/-PDV2E8Jn6k that resolves the massive amount of _grokparsefailure for the NIDS, I've not seen another log like that after the correction, not sure if related to the TODO "check for _grokparsefailure" thing.

[dougburks]

Hi @behkxyz ,

Yes, the fix in that google group thread was already in our dev repo and so it will automatically go into the next release. As we announce future releases, please continue to watch out for _grokparsefailures and report them on the mailing list so they can be resolved.

Thanks!

[dougburks]

submitted to QA group for testing:
https://groups.google.com/d/topic/security-onion-testing/Nn3jWvxhqk0/discussion

[dougburks]

published:
http://blog.securityonion.net/2017/09/elastic-stack-alpha-release-and.html