Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zeek 3.0.3 #1726

Closed
dougburks opened this issue Feb 26, 2020 · 12 comments
Closed

Zeek 3.0.3 #1726

dougburks opened this issue Feb 26, 2020 · 12 comments
Assignees
Projects

Comments

@dougburks
Copy link
Contributor

@dougburks dougburks commented Feb 26, 2020

@dougburks dougburks self-assigned this Feb 26, 2020
@dougburks dougburks added this to To do in 16.04.6.5 via automation Feb 26, 2020
@dougburks dougburks moved this from To do to In progress in 16.04.6.5 Mar 2, 2020
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 9, 2020

@dougburks dougburks changed the title Zeek 3.0.2 Zeek 3.0.3 Mar 9, 2020
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 10, 2020

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 11, 2020

List of packages to be tested:

  • securityonion-bro - 3.0.3-1ubuntu1securityonion1 (Zeek 3.0.3)
  • securityonion-bro-afpacket - 1.3.0-1ubuntu1securityonion21
  • securityonion-bro-scripts - 20121004-0ubuntu0securityonion104
  • securityonion-samples-bro - 20170824-1ubuntu1securityonion6

Other issues to be tested:
#1727

An overview of the testing process can be found in the comments below.

Please record all testing results via comments on this issue.

Thanks in advance for your time effort!

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 11, 2020

How To Start Testing

  • install the current 16.04 ISO image

  • snapshot the VM if possible

  • add the test PPA:

sudo add-apt-repository -y ppa:securityonion/test
  • update:
sudo soup

Please note that we also have Elastic packages and Docker images in testing right now, so if you want to test just this Zeek update, you should be able to replace that last command with:

sudo apt update && sudo apt install securityonion-bro securityonion-bro-afpacket securityonion-bro-scripts securityonion-samples-bro
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 11, 2020

How To Verify Proper Zeek Operation

  • first, please note that Bro has been renamed to Zeek but the packages still adhere to the traditional bro naming convention

  • as the Zeek packages install, verify that the package installation scripts display a message about checking configuration and adding back any local customizations

  • verify that Bro packages were upgraded:

dpkg -l |grep securityonion-bro
  • verify that the new Zeek packages create symlinks as necessary so that the new zeek paths resolve to the traditional bro locations (for example: /opt/zeek is a symlink to /opt/bro, /nsm/zeek is a symlink to /nsm/bro, etc.)

  • verify that the new Zeek packages create symlinks as necessary so that well known bro files resolve to the new zeek locations (for example: /opt/bro/etc/broctl.cfg is a symlink to zeekctl.cfg, so it can be accessed via /opt/bro/etc/broctl.cfg or /opt/zeek/etc/zeekctl.cfg)

  • if new installation, run through Setup

  • verify that the package installation scripts backed up the following with a _pre-3.0.3 extension:
    /opt/bro/etc/
    /opt/bro/share/bro/

  • verify that StatusCmdShowAll has been set to 0 in /opt/zeek/etc/zeekctl.cfg:

grep StatusCmdShowAll /opt/zeek/etc/zeekctl.cfg
  • verify that "lb_custom.InterfacePrefix=af_packet::" has been added to /opt/zeek/etc/zeekctl.cfg:
grep af_packet /opt/zeek/etc/zeekctl.cfg
  • Restart Zeek:
sudo so-zeek-restart
  • check status:
sudo so-status
  • check Zeek startup logs for any warnings/errors out of the ordinary:
cat /nsm/zeek/logs/current/reporter.log
cat /nsm/zeek/logs/current/stdout.log
cat /nsm/zeek/logs/current/stderr.log
  • replay LOTS of traffic:
sudo so-test
  • verify that files are extracted to /nsm/zeek/extracted:
ls -alh /nsm/zeek/extracted
  • verify that /nsm/zeek/logs/current/conn.log contains the proper sensorname at the end of each log entry:
cat /nsm/zeek/logs/current/conn.log
  • verify that Zeek logs are in the same format as they were pre-upgrade (should be JSON by default).

  • verify that the Elastic Stack is parsing and displaying Zeek logs properly (whether JSON or TSV format)

  • verify that you can pivot to CapMe for both TCP and UDP traffic

  • check sostat output for anything out of the ordinary (specifically, check the pf_ring and Zeek sections for packet loss)

  • verify that Zeek ja3 script is loaded and logging:

grep ja3 /nsm/zeek/logs/current/*
  • verify that Zeek hassh script is loaded and logging:
grep hassh /nsm/zeek/logs/current/*
  • verify that /etc/cron.d/bro has been moved to /etc/cron.d/zeek and that it works properly

  • verify that /opt/samples/zeek is a symlink to /opt/samples/bro

  • verify that everything else works properly with no regressions

  • reboot and make sure everything still works properly

Please test in as many different combinations as possible:

  • Evaluation Mode (Bro Standalone mode) vs Production Mode (Bro cluster mode)

  • single sniffing interface vs multiple sniffing interfaces

  • file extraction enabled or disabled

  • json-logs enabled or disabled

  • traffic without vlan tags vs traffic with vlan tags

  • new installation vs upgrade

  • Zeek cluster mode - PF_RING (lb_method=pf_ring) vs AF_PACKET (lb_method=custom)

@dougburks dougburks moved this from In progress to In Testing in 16.04.6.5 Mar 11, 2020
@petiepooo

This comment has been minimized.

Copy link

@petiepooo petiepooo commented Mar 11, 2020

Soaking on one system to verify no cpu load spiking... I'll try to get through some of the other test situations asap..

Thanks for turning this around so quickly, Doug!

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 12, 2020

Thanks @petiepooo , I really appreciate you bringing this issue to light and helping us turn it around quickly!

@petiepooo

This comment has been minimized.

Copy link

@petiepooo petiepooo commented Mar 13, 2020

Soaked on one system for nearly two days and another busier system for over 12 hours.. I would have normally seen at least one worker jump to 100% CPU by now, I believe, so I'm convinced this does fix the CPU load issue.

This is just using af_packet and 2 or 4 workers with the package run in as an upgrade. No regressions noted WRT extracts, elastic ingestion, ja3, hassh, etc. I haven't yet installed via ISO or tried eval mode, pf_ring, multiple ifaces, CSV logs, vlans, etc.

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 13, 2020

Thanks again for your time and testing @petiepooo !

@petiepooo

This comment has been minimized.

Copy link

@petiepooo petiepooo commented Mar 16, 2020

Soaked on 6 systems over the weekend with no ill effects and no spike in CPU usage. Still no regressions noted, but still limited to just upgrading existing deployments.

@weslambert

This comment has been minimized.

Copy link
Collaborator

@weslambert weslambert commented Mar 17, 2020

No issues with Zeek from my testing 👍 .

@dougburks dougburks moved this from In Testing to Tested in 16.04.6.5 Mar 17, 2020
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Mar 17, 2020

@dougburks dougburks closed this Mar 17, 2020
16.04.6.5 automation moved this from Tested to Done Mar 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
16.04.6.5
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.