Support import filter expressions #1742
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dominiklohmann
force-pushed
the
topic/import-filter-expressions
branch
from
5ff41e4 to
95bc328
Compare
We have long advertised this feature in the documentation of the import command, but never gotten around to implementing it. Turns out it really is quite simple to implement with the new `filter` function on table slices. Here's how you can use it: ```bash # Do not import Suricata.stats events as they are rather meaningless. vast import suricata '#type != "suricata.stats"' # Import only indicators to a matcher that have whose 'type' field is # 'ip' or 'ipv6' (requires the proprietary Live Matching plugin). vast matcher import -t my-ioc csv my-matcher 'type == "ip" || type == "ipv6"' ```
dominiklohmann
force-pushed
the
topic/import-filter-expressions
branch
from
95bc328 to
ca1940b
Compare
tobim
requested changes
Jul 6, 2021
Comment on lines
110
to
126
| const auto unfiltered_rows = slice.rows(); | ||
| if (self->state.filter) { | ||
| if (auto filtered_slice | ||
| = filter(std::move(slice), *self->state.filter)) { | ||
| VAST_VERBOSE("{} forwards {}/{} produced {} events after filtering", | ||
| self, filtered_slice->rows(), unfiltered_rows, | ||
| slice.layout().name()); | ||
| self->state.mgr->out().push(std::move(*filtered_slice)); | ||
| } else { | ||
| VAST_VERBOSE("{} forwards 0/{} produced {} events after filtering", | ||
| self, unfiltered_rows, slice.layout().name()); | ||
| } | ||
| } else { | ||
| VAST_VERBOSE("{} forwards {} produced {} events", self, | ||
| unfiltered_rows, slice.layout().name()); | ||
| self->state.mgr->out().push(std::move(slice)); | ||
| } |
There was a problem hiding this comment.
The log severity for all of those messages has to be DEBUG.
There was a problem hiding this comment.
Done. I also moved the logic to a function in the source state that is used for both source types.
tobim
approved these changes
Jul 9, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📔 Description
We have long advertised this feature in the documentation of the import command, but never gotten around to implementing it. Turns out it really is quite simple to implement with the new
filterfunction on table slices.Here's how you can use it:
📝 Checklist
🎯 Review Instructions
File-by-file. Give it a spin locally.