Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make JSON field selectors configurable #1974

Merged
merged 3 commits into from Nov 25, 2021

Conversation

dominiklohmann
Copy link
Member

@dominiklohmann dominiklohmann commented Nov 22, 2021

This adds a new --selector=<field>[:<prefix>] option to the JSON import. The field contains the event name, which when prefixed with the optional prefix is used for selecting the event type.

This previously existed as a hard-coded option for the Zeek Stremaing JSON and Suricata Eve JSON readers, which still exist unchanged.

The following commands are now equivalent:

# Import Suricata Eve JSON
vast import suricata
vast import json --selector=event_type:suricata

# Import Zeek Streaming JSON
vast import zeek-json
vast import json --selector=_path:zeek

馃摑 Checklist

  • All user-facing changes have changelog entries.
  • The changes are reflected on docs.tenzir.com/vast, if necessary.
  • The PR description contains instructions for the reviewer, if necessary.

馃幆 Review Instructions

File-by-file. I left some notes as review comments that are helpful to look at.

Note that this is based on #1888 to avoid merge conflicts.

@dominiklohmann dominiklohmann added the feature New functionality label Nov 22, 2021
@dominiklohmann dominiklohmann requested a review from a team November 22, 2021 14:05
This adds a new `--selector=<field>[:<prefix>]` option to the JSON
import. The *field* contains the *event name*, which when prefixed with
the optional *prefix* is used for selecting the event type.

This previously existed as a hard-coded option for the Zeek Stremaing
JSON and Suricata Eve JSON readers, which still exist unchanged.

The following commands are now equivalent:

```sh
# Import Suricata Eve JSON
vast import suricata
vast import json --selector=event_type:suricata

# Import Zeek Streaming JSON
vast import zeek-json
vast import json --selector=_path:zeek
```
@dominiklohmann dominiklohmann force-pushed the story/sc-29697/import-json-selector branch from c09eee5 to e081d9c Compare November 22, 2021 14:12
@dominiklohmann dominiklohmann linked an issue Nov 22, 2021 that may be closed by this pull request
@mavam
Copy link
Member

mavam commented Nov 23, 2021

How could I interpret this with the upcoming schema module changes? Would prefix just be the module?

@dominiklohmann
Copy link
Member Author

How could I interpret this with the upcoming schema module changes? Would prefix just be the module?

Yes, exactly.

Copy link
Contributor

@6yozo 6yozo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

libvast/vast/format/json/field_selector.hpp Show resolved Hide resolved
Before this change, running `vast spawn import json --selector=:`
allowed users to crash the connected to VAST server.
Base automatically changed from story/ch20439/type-flatbuffers to master November 25, 2021 10:47
@dominiklohmann dominiklohmann merged commit e13369d into master Nov 25, 2021
@dominiklohmann dominiklohmann deleted the story/sc-29697/import-json-selector branch November 25, 2021 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
3 participants