New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic Stack Alpha Release #1095

Closed
dougburks opened this Issue Apr 12, 2017 · 21 comments

Comments

Projects
None yet
9 participants
@dougburks
Contributor

dougburks commented Apr 12, 2017

  • Elasticsearch

    • Docker container
    • run as a non-root user
    • log to /var/log/elasticsearch/
    • add script to reset Elasticsearch database
    • --env "bootstrap_memory_lock=true"
    • --ulimit memlock=-1:-1
    • --env ES_JAVA_OPTS="-Xms$ELASTICSEARCH_HEAP -Xmx$ELASTICSEARCH_HEAP"
    • set initial $ELASTICSEARCH_HEAP based on available RAM
    • MAX_OPEN_FILES=65536
    • elasticsearch - nofile 65536
    • elasticsearch soft/hard nproc 2048
    • field [include_in_all] is deprecated, as [_all] is deprecated, and will be disallowed in 6.0, use [copy_to] instead.
  • Kibana

    • Docker container
    • run as a non-root user
    • log to /var/log/kibana/
    • set source_ip and destination_ip fields to type "ip"
    • dashboard for each Bro log
    • resolve field type conflict for status_code field
    • remove ELSA shortcut from desktop and replace with Kibana
    • Top Source/Destination by Total Bytes
    • add source port to NIDS dashboard
    • configure Kibana index for 1 shard and 0 replicas
    • Indicator dashboard should only search last 24 hours by default
    • Tables should show at least 10 results (perhaps 100)
    • add visualizations and dashboard for Bro syslog.log
    • add visualizations and dashboard for standard syslog coming from syslog-ng
    • DNS dashboard needs to be adjusted for recent navigation menu change (scroll bar)
    • comment out unnecessary config in /etc/kibana/kibana.yml
    • build new Kibana image with links to Squert and Logout in the side bar
    • add message visualization to Notices dashboard
    • add sub_message visualization to Notices dashboard
    • DNS query field should hyperlink to Indicator dashboard
    • SSL Server Name field should hyperlink to Indicator dashboard
    • remove Squert and Logout links from Navigation visualization in favor of new links in side bar
    • HTTP site (virtual_host) field should hyperlink to Indicator dashboard
    • Intel indicator field should hyperlink to Indicator dashboard
    • Bro Files - Source IP field should hyperlink to Indicator dashboard
    • move ElastAlert menu entry from Other to Alert Data
    • add dashboard for ElastAlert
    • fix index-pattern-field(s) for Sysmon
    • fix index-pattern-field(s) for Autoruns
    • source_port field should hyperlink to Indicator dashboard
    • destination_port field should hyperlink to Indicator dashboard
    • set timepicker:timeDefaults to search last 24 hours
    • set dashboard:defaultDarkTheme to true
    • Bro Connections maps are showing a scroll bar for the navigation menu, needs resizing
    • Firewall dashboard has multiple "could not locate that index-pattern-field" errors
  • Logstash

    • Docker container
    • run as a non-root user
    • log to /var/log/logstash/
    • fix typo in filename 0001_intput_json.conf
    • add filters for bro_modbus, bro_pe, bro_rfb, bro_sip, bro_radius, bro_dnp3
    • change event to event.set and event.get where necessary
    • only add bro tag if type is bro
    • move rename from 0000_input to 1000_preprocess
    • comment out 0003_input_syslog.conf
    • update Bro patterns for Bro 2.5 (#1029)
    • add parser for OSSEC alerts
    • add parsers for Bro SMB
    • parse Bro Notice fields message and sub_message as msg and sub_msg
    • retain IPv6 addresses
    • add logstash_time field
    • persistent queues
    • set pipeline.workers: 1
    • NIDS alert - lookup rule in /etc/nsm/rules/ and insert into alert
    • parse pfSense firewall logs
    • parse Sysmon logs
    • parse Autoruns logs
    • rebuild logstash image with new logstash-filter-rest plugin
    • update pfSense parser with fix for grokparsefailure on ICMPv6 traffic
    • update Snort parser to parse sensor name
  • Curator

    • Docker container
    • run as a non-root user
    • log to /var/log/curator/
  • ElastAlert

    • Docker container
    • run as a non-root user
    • log to /var/log/elastalert/
    • disable elasticsearch replicas for elastalert_status and reduce shards to 1
    • default rules should use use_count_query
  • Freqserver

    • Docker container
    • run as a non-root user
  • Domainstats

    • Docker container
    • run as a non-root user
  • so-elastic-configure

    • check for options in /etc/nsm/securityonion.conf before adding them
  • so-elastic-configure-elastalert

    • check to see if elastalert_status index exists before deleting
  • so-elastic-start

  • so-start / so-stop / so-restart

    • create small wrapper scripts that control both NSM and Elastic
    • create /usr/sbin/so-status
  • sostat

    • provide stats for Elastic stack
  • soup

    • when docker is upgraded, it stops all containers, need to restart
  • ELSA

    • dump ELSA data out to filesystem in a form that Logstash can read
    • have Logstash read the original Bro timestamp and use that for ingestion into ES
  • CapMe

    • move to single sign-on
    • update for cross cluster search
  • Sguil

    • change ELSA pivot to Kibana search for that IP on Indicator dashboard
  • Squert

    • pivot from IP address to Kibana search for that IP on Indicator dashboard
    • move to single sign-on
  • Setup

    • add Elastic option to Setup for new installations but mark it Experimental
    • exclude docker interfaces from interface selection
  • OSSEC

    • rootcheck is alerting on Files hidden inside directory '/var/lib/docker/aufs
    • opening multiple kibana tabs may result in active response "High amount of POST requests in a small period of time (likely bot)", check for /es_admin
  • distributed deployments

  • update existing packages

    • securityonion-setup
    • securityonion-sostat
  • build new packages where necessary

    • securityonion-elastic
    • securityonion-samples-bro
  • develop tests for QA

    • check for _grokparsefailure, _csvparsefailure, _rubyexception
    • check log files:
      • /var/log/elasticsearch/
      • /var/log/logstash/
      • /var/log/kibana/
      • /var/log/elastalert/
      • /var/log/curator/
      • /var/log/freqserver/
      • /var/log/apache2/*
@JonZeolla

This comment has been minimized.

Show comment
Hide comment
@JonZeolla

JonZeolla Jun 4, 2017

Not sure if it's helpful, but not too long back I put together a wiki article covering how to tune ES, focused on bro data for another project. It is intended for 2.4, and I noticed you recently upgraded to 5.4, so I'm not sure if this is still valid.

Not sure if it's helpful, but not too long back I put together a wiki article covering how to tune ES, focused on bro data for another project. It is intended for 2.4, and I noticed you recently upgraded to 5.4, so I'm not sure if this is still valid.

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Jun 4, 2017

Contributor

Thanks @JonZeolla !

Contributor

dougburks commented Jun 4, 2017

Thanks @JonZeolla !

@ragdelaed

This comment has been minimized.

Show comment
Hide comment
@ragdelaed

ragdelaed Jun 6, 2017

are there any plans for a hadoop/big data enabled back end for SO?
or do you plan to use kafka, storm, zeppelin or any other tools from the hadoop ecosystem?
also any profiling, baseling, or anything like that for statistical or machine learning analysis?
and do you plan to pull in windows event logs in the future in addition to bro/snort/etc?

thanks for your work on this, its an excellent product, elegant, and it fits in my brain nicely. :)

are there any plans for a hadoop/big data enabled back end for SO?
or do you plan to use kafka, storm, zeppelin or any other tools from the hadoop ecosystem?
also any profiling, baseling, or anything like that for statistical or machine learning analysis?
and do you plan to pull in windows event logs in the future in addition to bro/snort/etc?

thanks for your work on this, its an excellent product, elegant, and it fits in my brain nicely. :)

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Jun 6, 2017

Contributor

Hi @ragdelaed ,

You can pull in Windows Event Logs today using OSSEC agent or several other event log collectors.

Right now, our main focus is integrating the Elastic stack. Once that is done, we'll look at what makes the most sense for our next project.

Thanks!

Contributor

dougburks commented Jun 6, 2017

Hi @ragdelaed ,

You can pull in Windows Event Logs today using OSSEC agent or several other event log collectors.

Right now, our main focus is integrating the Elastic stack. Once that is done, we'll look at what makes the most sense for our next project.

Thanks!

@mv003348

This comment has been minimized.

Show comment
Hide comment
@mv003348

mv003348 Jun 13, 2017

@dougburks - This migration is AWESOME. I have already taken version one if your script and turned it into a fully functioning AWS AMI. As a proof of concept 1:1 ratio (management to sniffing interface), I have collected data to show my teams what type of traffic is coming into our system. I have begun to play around with ElastAlert, but running into some index pattern issues. I have showed this off to my teams enough times that once the migration to ELK is completed, we are going to be implementing this across the entire company. All I can say is JOB WELL DONE SIR and THANK YOU!

FYI - I am hoping to be at the Security Onion conference this year. Very excited!

@dougburks - This migration is AWESOME. I have already taken version one if your script and turned it into a fully functioning AWS AMI. As a proof of concept 1:1 ratio (management to sniffing interface), I have collected data to show my teams what type of traffic is coming into our system. I have begun to play around with ElastAlert, but running into some index pattern issues. I have showed this off to my teams enough times that once the migration to ELK is completed, we are going to be implementing this across the entire company. All I can say is JOB WELL DONE SIR and THANK YOU!

FYI - I am hoping to be at the Security Onion conference this year. Very excited!

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Jun 13, 2017

Contributor

Hi @mv003348 ,

Thanks for the feedback! Hope to see you at the conference!

Contributor

dougburks commented Jun 13, 2017

Hi @mv003348 ,

Thanks for the feedback! Hope to see you at the conference!

@gaganova

This comment has been minimized.

Show comment
Hide comment
@gaganova

gaganova Jun 30, 2017

Is this transition to elastic stack something that will definitely happen? Why? I would be very interested in reading about your reasons for this decision. Don't you think ES is too much bloat?

Is this transition to elastic stack something that will definitely happen? Why? I would be very interested in reading about your reasons for this decision. Don't you think ES is too much bloat?

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Jun 30, 2017

Contributor

Hi @gaganova ,

This is most likely going to happen. Our community and our customers have been asking for the Elastic Stack for quite some time and we try to be responsive to demand.

Contributor

dougburks commented Jun 30, 2017

Hi @gaganova ,

This is most likely going to happen. Our community and our customers have been asking for the Elastic Stack for quite some time and we try to be responsive to demand.

@gaganova

This comment has been minimized.

Show comment
Hide comment
@gaganova

gaganova Jun 30, 2017

I am asking myself if people demand that because they have only seen nice ELK screenshots or if they have actually used that bloated java stack in real life. Of course there is no answer to that, so hopefully this project will survive this decision. Good Luck! Anybody interested in maintaining the ELSA based version with 16.04?

I am asking myself if people demand that because they have only seen nice ELK screenshots or if they have actually used that bloated java stack in real life. Of course there is no answer to that, so hopefully this project will survive this decision. Good Luck! Anybody interested in maintaining the ELSA based version with 16.04?

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Jun 30, 2017

Contributor

Hi @gaganova ,

Replies inline.

I am asking myself if people demand that because they have only seen nice ELK screenshots or if they have actually used that bloated java stack in real life. Of course there is no answer to that, so hopefully this project will survive this decision. Good Luck!

I can tell you that there are many folks in our community who have already rolled their own Elastic deployment and are quite happy with it.

Anybody interested in maintaining the ELSA based version with 16.04?

While ELSA certainly has low resource requirements, that results in quite a few drawbacks which limit hunting capabilities (only 12 indexed fields, no multiple groupby, just to name a few). These limitations caused the lead developer of ELSA to then migrate to Elasticsearch for what was going to be ELSA 2.0. The future of ELSA 2.0 is currently unclear, thus we are moving to the Elastic stack.

Contributor

dougburks commented Jun 30, 2017

Hi @gaganova ,

Replies inline.

I am asking myself if people demand that because they have only seen nice ELK screenshots or if they have actually used that bloated java stack in real life. Of course there is no answer to that, so hopefully this project will survive this decision. Good Luck!

I can tell you that there are many folks in our community who have already rolled their own Elastic deployment and are quite happy with it.

Anybody interested in maintaining the ELSA based version with 16.04?

While ELSA certainly has low resource requirements, that results in quite a few drawbacks which limit hunting capabilities (only 12 indexed fields, no multiple groupby, just to name a few). These limitations caused the lead developer of ELSA to then migrate to Elasticsearch for what was going to be ELSA 2.0. The future of ELSA 2.0 is currently unclear, thus we are moving to the Elastic stack.

@mattraneri

This comment has been minimized.

Show comment
Hide comment
@mattraneri

mattraneri Jul 5, 2017

Hi @dougburks,

First and foremost, love this project. I would love to add the ELK stack to my deployment, but is there a timeline of when it will officially be supported? I'd hate to implement this in our deployment and have the official project roll out ELK two weeks later. Just asking because the last blog post was all I was able to find directly referencing the script, which is now months old.

Thank you!

Hi @dougburks,

First and foremost, love this project. I would love to add the ELK stack to my deployment, but is there a timeline of when it will officially be supported? I'd hate to implement this in our deployment and have the official project roll out ELK two weeks later. Just asking because the last blog post was all I was able to find directly referencing the script, which is now months old.

Thank you!

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Jul 5, 2017

Contributor

Hi @mattraneri ,

We've had a few blog posts about the Elastic integration:
http://blog.securityonion.net/search/label/elastic

We're not ready to commit to any timelines at this point. This is a large project and we still have a lot of work left to do. This is an important project and we want to make sure that it's done right!

Thanks!

Contributor

dougburks commented Jul 5, 2017

Hi @mattraneri ,

We've had a few blog posts about the Elastic integration:
http://blog.securityonion.net/search/label/elastic

We're not ready to commit to any timelines at this point. This is a large project and we still have a lot of work left to do. This is an important project and we want to make sure that it's done right!

Thanks!

@mv003348

This comment has been minimized.

Show comment
Hide comment
@mv003348

mv003348 Aug 8, 2017

@dougburks - I have been following the updates and launching new EC2 instances every time a new TP comes out. All I can say is, :mind blown:. This stuff is AWESOME! Whenever it becomes a stable version, I am making it our Enterprise IDS tool...all I need now are boat loads of SO stickers! THANK YOU!

mv003348 commented Aug 8, 2017

@dougburks - I have been following the updates and launching new EC2 instances every time a new TP comes out. All I can say is, :mind blown:. This stuff is AWESOME! Whenever it becomes a stable version, I am making it our Enterprise IDS tool...all I need now are boat loads of SO stickers! THANK YOU!

@kxuanCeltik

This comment has been minimized.

Show comment
Hide comment
@kxuanCeltik

kxuanCeltik Aug 23, 2017

Hi @dougburks,

Congratulations, this integration is awesome.

Where are the dockerfiles o how manage the containers? How is the deployment ?

THANKS !!!

Hi @dougburks,

Congratulations, this integration is awesome.

Where are the dockerfiles o how manage the containers? How is the deployment ?

THANKS !!!

@kxuanCeltik

This comment has been minimized.

Show comment
Hide comment
@kxuanCeltik

kxuanCeltik Aug 24, 2017

Sorry I've already found docker images.

But I try config with Production Mode and I have problems with dockers.

I would like test the integration to do an academic work,
maybe, Could I open other issues?

Sorry I've already found docker images.

But I try config with Production Mode and I have problems with dockers.

I would like test the integration to do an academic work,
maybe, Could I open other issues?

@weslambert

This comment has been minimized.

Show comment
Hide comment
@weslambert

weslambert Aug 24, 2017

Collaborator

@kxuanCeltik ,

Please see: http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html

This technology preview only works with Evaluation Mode currently.

Instead of opening an issue, or commenting on this issue, please use our mailing list:

https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists#security-onionv

Also see: https://groups.google.com/forum/#!topic/security-onion/Sr8wrJSQxo0

Thanks,
Wes

Collaborator

weslambert commented Aug 24, 2017

@kxuanCeltik ,

Please see: http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html

This technology preview only works with Evaluation Mode currently.

Instead of opening an issue, or commenting on this issue, please use our mailing list:

https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists#security-onionv

Also see: https://groups.google.com/forum/#!topic/security-onion/Sr8wrJSQxo0

Thanks,
Wes

@kxuanCeltik

This comment has been minimized.

Show comment
Hide comment

Ok, thanks.

@Behkxyz

This comment has been minimized.

Show comment
Hide comment
@Behkxyz

Behkxyz Aug 24, 2017

@dougburks just to add some notes https://groups.google.com/forum/#!topic/security-onion/-PDV2E8Jn6k that resolves the massive amount of _grokparsefailure for the NIDS, I've not seen another log like that after the correction, not sure if related to the TODO "check for _grokparsefailure" thing.

Behkxyz commented Aug 24, 2017

@dougburks just to add some notes https://groups.google.com/forum/#!topic/security-onion/-PDV2E8Jn6k that resolves the massive amount of _grokparsefailure for the NIDS, I've not seen another log like that after the correction, not sure if related to the TODO "check for _grokparsefailure" thing.

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

dougburks Aug 24, 2017

Contributor

Hi @Behkxyz ,

Yes, the fix in that google group thread was already in our dev repo and so it will automatically go into the next release. As we announce future releases, please continue to watch out for _grokparsefailures and report them on the mailing list so they can be resolved.

Thanks!

Contributor

dougburks commented Aug 24, 2017

Hi @Behkxyz ,

Yes, the fix in that google group thread was already in our dev repo and so it will automatically go into the next release. As we announce future releases, please continue to watch out for _grokparsefailures and report them on the mailing list so they can be resolved.

Thanks!

@dougburks dougburks changed the title from Elastic Stack to Elastic Stack Alpha Release Sep 6, 2017

@dougburks

This comment has been minimized.

Show comment
Hide comment
@dougburks

This comment has been minimized.

Show comment
Hide comment

@dougburks dougburks closed this Sep 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment