Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

securityonion-elastic: update parsers for Zeek 3 #1680

Closed
dougburks opened this issue Dec 23, 2019 · 6 comments
Closed

securityonion-elastic: update parsers for Zeek 3 #1680

dougburks opened this issue Dec 23, 2019 · 6 comments
Assignees
Projects

Comments

@dougburks
Copy link
Contributor

@dougburks dougburks commented Dec 23, 2019

As we migrate to Zeek 3, some log formats have changed, so we need to update our parsers accordingly:

  • remove orig_cc,resp_cc,original_country_code, and respond_country_code from conn (#1630)
  • add origin to http
  • remove dropped from notice
  • change remote_ip to tunnel_client (string) in radius
  • add client_channels to rdp

This also means that we need to add the following fields to the template:

  • client_channels
  • origin
  • tunnel_client

Reference the Zeek 3.0.0 section here:
https://github.com/zeek/zeek/blob/release/NEWS

We need to make sure all 3 parsing pipelines are updated and tested:

  • traditional TSV via Logstash parsing
  • JSON via Logstash parsing
  • JSON via Elasticsearch ingest node parsing
@dougburks dougburks added this to To do in 16.04.6.4 via automation Dec 23, 2019
@dougburks dougburks moved this from To do to In progress in 16.04.6.4 Dec 23, 2019
@dougburks dougburks self-assigned this Dec 23, 2019
@dougburks dougburks changed the title securityonion-elastic: update CSV/TSV parsers for Zeek 3 securityonion-elastic: update parsers for Zeek 3 Dec 23, 2019
dougburks added a commit to Security-Onion-Solutions/securityonion-elastic that referenced this issue Dec 24, 2019
@dougburks dougburks moved this from In progress to In Testing in 16.04.6.4 Jan 4, 2020
@defensivedepth

This comment has been minimized.

Copy link
Collaborator

@defensivedepth defensivedepth commented Jan 9, 2020

JSON via Logstash parsing -


image

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Jan 9, 2020

Hmm...client_channels should have been added to logstash_template.json:
Security-Onion-Solutions/securityonion-elastic@6f2384f

Perhaps you're looking at an existing index that was created with the old template and the next day's index will be created with the new template?

@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Jan 10, 2020

Looking into this further, we might need to update the Kibana Index Pattern for *:logstash-*.

dougburks added a commit to Security-Onion-Solutions/securityonion-elastic that referenced this issue Jan 14, 2020
@dougburks

This comment has been minimized.

Copy link
Contributor Author

@dougburks dougburks commented Jan 14, 2020

securityonion-elastic - 20190510-1ubuntu1securityonion83 is now available at ppa:securityonion/test and adds client_channels, origin, and tunnel_client to the Kibana Index Pattern.

@weslambert

This comment has been minimized.

Copy link
Collaborator

@weslambert weslambert commented Jan 17, 2020

Looks good 👍

@weslambert weslambert moved this from In Testing to Tested in 16.04.6.4 Jan 17, 2020
@dougburks

This comment has been minimized.

@dougburks dougburks closed this Feb 5, 2020
16.04.6.4 automation moved this from Tested to Done Feb 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
16.04.6.4
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.