Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Zeek 3.0.1, Elastic 6.8.6, and related updates #1691

Closed
dougburks opened this issue Dec 24, 2019 · 8 comments
Closed

Test Zeek 3.0.1, Elastic 6.8.6, and related updates #1691

dougburks opened this issue Dec 24, 2019 · 8 comments
Projects

Comments

@dougburks
Copy link
Contributor

@dougburks dougburks commented Dec 24, 2019

List of packages to be tested:

  • securityonion-bro - 3.0.1-1ubuntu1securityonion10 (Zeek 3.0.1)
  • securityonion-bro-afpacket - 1.3.0-1ubuntu1securityonion17
  • securityonion-bro-scripts - 20121004-0ubuntu0securityonion100
  • securityonion-elastic - 20190510-1ubuntu1securityonion83
  • securityonion-nsmnow-admin-scripts - 20120724-0ubuntu0securityonion225
  • securityonion-onionsalt - 20140917-0ubuntu0securityonion28
  • securityonion-samples-bro - 20170824-1ubuntu1securityonion4
  • securityonion-setup - 20120912-0ubuntu0securityonion325
  • securityonion-sostat - 20120722-0ubuntu0securityonion141
  • securityonion-tcpudpflow - 001-0ubuntu0securityonion10
  • securityonion-web-page - 20141015-0ubuntu0securityonion105

List of Docker images to be tested:

  • Elasticsearch 6.8.6 (both OSS and Elastic Features versions)
  • Logstash 6.8.6 (both OSS and Elastic Features versions)
  • Kibana 6.8.6 (both OSS and Elastic Features versions)
  • Freqserver
  • Domainstats
  • ElastAlert
  • Curator

An overview of the testing process can be found in the comments below. For a complete list of Issues to be tested, please see the Testing/Tested columns at https://github.com/Security-Onion-Solutions/security-onion/projects/10

Please record all testing results via comments on this issue or the individual issues at https://github.com/Security-Onion-Solutions/security-onion/projects/10

Thanks in advance for your time effort!

@dougburks dougburks added this to To do in 16.04.6.4 via automation Dec 24, 2019
@dougburks dougburks changed the title Test Zeek, NSM, Setup, and Elastic Test Zeek, NSM, Setup, tcpudpflow, and Elastic Dec 26, 2019
@dougburks dougburks changed the title Test Zeek, NSM, Setup, tcpudpflow, and Elastic Test Zeek 3.0.1, Elastic 6.8.6, and related updates Dec 26, 2019
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Jan 4, 2020

How To Start Testing

  • install the current 16.04 ISO image

  • snapshot the VM if possible

  • add the test PPA:

sudo add-apt-repository -y ppa:securityonion/test
  • change DOCKERHUB from "securityonionsolutions" to "securityonionsolutionstest" (OSS license):
sudo sed -i 's|DOCKERHUB="securityonionsolutions"|DOCKERHUB="securityonionsolutionstest"|g' /etc/nsm/elasticdownload.conf
  • update:
sudo soup
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Jan 4, 2020

How To Verify Proper Zeek Operation

  • first, please note that Bro has been renamed to Zeek but the packages still adhere to the traditional bro naming convention

  • as the Zeek packages install, verify that the package installation scripts display a message about checking configuration and adding back any local customizations

  • verify that Bro packages were upgraded:

dpkg -l |grep securityonion-bro
  • verify that the new Zeek packages create symlinks as necessary so that the new zeek paths resolve to the traditional bro locations (for example: /opt/zeek is a symlink to /opt/bro, /nsm/zeek is a symlink to /nsm/bro, etc.)

  • verify that the new Zeek packages create symlinks as necessary so that well known bro files resolve to the new zeek locations (for example: /opt/bro/etc/broctl.cfg is a symlink to zeekctl.cfg, so it can be accessed via /opt/bro/etc/broctl.cfg or /opt/zeek/etc/zeekctl.cfg)

  • if new installation, run through Setup

  • verify that the package installation scripts backed up the following with a _pre-3.0.1 extension:
    /opt/bro/etc/
    /opt/bro/share/bro/

  • verify that StatusCmdShowAll has been set to 0 in /opt/zeek/etc/zeekctl.cfg:

grep StatusCmdShowAll /opt/zeek/etc/zeekctl.cfg
  • verify that "lb_custom.InterfacePrefix=af_packet::" has been added to /opt/zeek/etc/zeekctl.cfg:
grep af_packet /opt/zeek/etc/zeekctl.cfg
  • Restart Zeek:
sudo so-zeek-restart
  • check status:
sudo so-status
  • check Zeek startup logs for any warnings/errors out of the ordinary:
cat /nsm/zeek/logs/current/reporter.log
cat /nsm/zeek/logs/current/stdout.log
cat /nsm/zeek/logs/current/stderr.log
  • replay LOTS of traffic:
sudo so-test
  • verify that files are extracted to /nsm/zeek/extracted:
ls -alh /nsm/zeek/extracted
  • verify that /nsm/zeek/logs/current/conn.log contains the proper sensorname at the end of each log entry:
cat /nsm/zeek/logs/current/conn.log
  • verify that Zeek logs are in the same format as they were pre-upgrade (should be JSON by default).

  • verify that the Elastic Stack is parsing and displaying Zeek logs properly (whether JSON or TSV format)

  • verify that you can pivot to CapMe for both TCP and UDP traffic

  • check sostat output for anything out of the ordinary (specifically, check the pf_ring and Zeek sections for packet loss)

  • verify that Zeek ja3 script is loaded and logging:

grep ja3 /nsm/zeek/logs/current/*
  • verify that Zeek hassh script is loaded and logging:
grep hassh /nsm/zeek/logs/current/*
  • verify that /etc/cron.d/bro has been moved to /etc/cron.d/zeek and that it works properly

  • verify that everything else works properly with no regressions

  • reboot and make sure everything still works properly

Please test in as many different combinations as possible:

  • Evaluation Mode (Bro Standalone mode) vs Production Mode (Bro cluster mode)

  • single sniffing interface vs multiple sniffing interfaces

  • file extraction enabled or disabled

  • json-logs enabled or disabled

  • traffic without vlan tags vs traffic with vlan tags

  • new installation vs upgrade

  • Zeek cluster mode - PF_RING (lb_method=pf_ring) vs AF_PACKET (lb_method=custom)

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Jan 4, 2020

How To Verify Proper Elastic Operation

Please test in as many different combinations as possible:

  • verify that traditional Logstash parsing correctly parses Zeek logs in JSON or TSV format

  • verify that Elasticsearch Ingest parsing correctly parses Zeek logs in JSON format

  • verify Kibana dashboards visualize those parsed logs correctly

  • check elasticsearch, logstash, and kibana logs in /var/log/ for anything out of the ordinary

  • so-import-pcap vs sosetup-minimal vs traditional Setup

  • Setup GUI vs CLI

  • Evaluation Mode vs Production Mode

  • standalone vs distributed deployments

  • new installation vs upgrade

  • Elastic OSS vs Elastic Features license (use so-elastic-features to switch from OSS to Features)

  • SSO vs Elastic native auth (use so-elastic-auth to switch to Elastic native auth)

@dougburks dougburks moved this from To do to In Testing in 16.04.6.4 Jan 4, 2020
@weslambert

This comment has been minimized.

Copy link
Collaborator

@weslambert weslambert commented Jan 17, 2020

Tested in various configurations without any issues.

@bryant-treacle

This comment has been minimized.

Copy link

@bryant-treacle bryant-treacle commented Jan 31, 2020

broctl & zeekctl script in /usr/sbin/ not functioning properly. No text being displayed. I can still launch them from their native directory.
It looks like an issue is in the grep -v after we call the function. It work fine when I comment it out.

Everything else seems to work without issues.

@chris-cuevas

This comment has been minimized.

Copy link

@chris-cuevas chris-cuevas commented Feb 4, 2020

No issues seen in my testing with more than 70,000,000 events per hour.

As per the checklist for testing...

dpkg -l |grep securityonion-bro
ii securityonion-bro 3.0.1-1ubuntu1securityonion10 amd64 The Bro Network Security Monitor
ii securityonion-bro-afpacket 1.3.0-1ubuntu1securityonion17 all Plugin providing native AF_Packet support for Bro.
ii securityonion-bro-scripts 20121004-0ubuntu0securityonion100 all Bro scripts for Security Onion

root@test-host1:~# ls -l /opt/zeek
lrwxrwxrwx 1 root root 3 Feb 4 13:06 /opt/zeek -> bro

root@test-host1:~# ls -l /nsm/zeek
lrwxrwxrwx 1 root root 3 Feb 4 13:06 /nsm/zeek -> bro

root@test-host1:~# ls -l /opt/bro/etc/broctl.cfg
lrwxrwxrwx 1 root root 11 Feb 4 13:06 /opt/bro/etc/broctl.cfg -> zeekctl.cfg

root@test-host1:~# ls -l /opt/bro/
total 0
drwxr-xr-x 2 root root 257 Feb 4 13:06 bin
drwxr-xr-x 2 root root 101 Feb 4 13:13 etc
drwxr-xr-x 2 root root 60 Sep 17 18:26 etc_pre-2.6.4
drwxr-xr-x 2 root root 60 Feb 4 13:04 etc_pre-3.0.1

root@test-host1:~# ls -l /opt/bro/share/
total 0
lrwxrwxrwx 1 root root 4 Feb 4 13:06 bro -> zeek
drwxr-xr-x 4 root root 31 Feb 4 13:06 bro.pre-3.0.1

root@test-host1:~# grep StatusCmdShowAll /opt/zeek/etc/zeekctl.cfg
StatusCmdShowAll = 0

root@test-host1:~# grep af_packet /opt/zeek/etc/zeekctl.cfg
lb_custom.InterfacePrefix=af_packet::

root@test-host1:~# ls /etc/cron.d/
anacron capme mdadm netsniff-sync nsm-watchdog php salt-update sensor-clean sensor-newday sguil-db-purge so-sensor-backup-config so-server-backup-config squert-ip2c sysstat zeek

I have rebooted the system and things come up smoothly on reboot.

I had no issue with with zeekctl or broctl displaying text but I only ran a zeekctl config so didn't test that extensively.

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Feb 4, 2020

Thanks @chris-cuevas !

@dougburks

This comment has been minimized.

@dougburks dougburks closed this Feb 5, 2020
16.04.6.4 automation moved this from In Testing to Done Feb 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
16.04.6.4
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.