Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support json-delimited import for Zeek logs #65

Closed
mmguero opened this issue Nov 12, 2021 · 2 comments
Closed

support json-delimited import for Zeek logs #65

mmguero opened this issue Nov 12, 2021 · 2 comments
Assignees
@mmguero
Labels
enhancement New feature or request logstash Relating to Malcolm's use of Logstash upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek
Projects
Malcolm
Milestone
v24.03.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Nov 12, 2021

Currently Malcolm only supports the standard tab-delimited format for Zeek logs. There have been some requests to import JSON format as well.
@mmguero mmguero added enhancement New feature or request logstash Relating to Malcolm's use of Logstash upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek labels Nov 12, 2021
@mmguero mmguero added this to To do in Malcolm Nov 17, 2021
@mmguero
Copy link
Collaborator Author

mmguero commented Dec 19, 2023

What this would entail:

  • detecting somewhere towards the beginning of the zeek pipeline the JSON vs. TSV format and parsing it accordingly
  • for individual logs types, renaming fields such that they match what we're generating from the TSV-parsed logs (search for _field_names in that file)

@mmguero mmguero added this to the v24.02.0 milestone Jan 2, 2024
@mmguero mmguero modified the milestones: v24.02.0, staging Jan 15, 2024
@mmguero mmguero modified the milestones: z.staging, v24.03.0 Feb 15, 2024
@mmguero mmguero self-assigned this Feb 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 16, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); almost certainly br…
bf232c6 
…oken at this point
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 16, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); almost certainly br…
cea233a 
…oken at this point
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 20, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); almost certainly br…
cc90ca9 
…oken at this point
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 20, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); almost certainly br…
02898f8 
…oken at this point
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); almost certainly br…
0879cea 
…oken at this point
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); almost certainly br…
26f7238 
…oken at this point
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
096e32c
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
c2826f8
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
540c0eb
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
5476493
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
d9021c9
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
88d8147
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 22, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
594c641
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 22, 2024
@mmguero
for supporting JSON logs from Zeek (idaholab#65); getting closer
1e57a3a
@mmguero
Copy link
Collaborator Author

mmguero commented Feb 22, 2024

As far as I can tell this is done. Everything seems to be working. Will reopen (or log another issue) if I find anything else.

@mmguero mmguero closed this as completed Feb 22, 2024
This was referenced Mar 4, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
enhancement New feature or request logstash Relating to Malcolm's use of Logstash upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek
Projects
Malcolm
Status: Released
Malcolm
  
To do
Milestone
v24.03.0
Development

No branches or pull requests
1 participant
@mmguero