Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement model substitution #1185

Merged
merged 16 commits into from Dec 1, 2020
Merged

Implement model substitution #1185

merged 16 commits into from Dec 1, 2020

Conversation

tobim
Copy link
Member

@tobim tobim commented Nov 19, 2020

馃摂 Description

This adds support for models to the resolve function of taxonomies. It can substitute records with either named or unnamed fields.

Notable supporting changes:

  • Model Definitions are now a single list containing both concepts and other models
  • The for_each_predicate helper has been updated to support failure.
  • records can now contain multiple fields of the same name

馃摑 Checklist

  • All user-facing changes have changelog entries.
  • The changes are reflected on docs.tenzir.com/vast, if necessary.
  • The PR description contains instructions for the reviewer, if necessary.
  • Add unit tests
  • Add integration tests

馃幆 Review Instructions

It's probably best to go commit-by-commit, but be aware that later commits already contain some naming improvements and functional extensions. The "implement Model substitution" contains the change for the model definitions.

@tobim tobim added the feature New functionality label Nov 19, 2020
@tobim tobim force-pushed the tdhtf/ch1724/model-substitution branch 3 times, most recently from 66abf30 to c90c170 Compare November 24, 2020 20:01
@tobim tobim marked this pull request as ready for review November 26, 2020 05:02
@tobim tobim marked this pull request as draft November 26, 2020 05:35
@tobim tobim force-pushed the tdhtf/ch1724/model-substitution branch from c90c170 to 3a0194e Compare November 26, 2020 07:34
@tobim tobim marked this pull request as ready for review November 26, 2020 07:53
@dominiklohmann
Copy link
Member

Nit: Can you move 6df0395 into a separate PR?

@tobim
Copy link
Member Author

tobim commented Nov 26, 2020

Nit: Can you move 6df0395 into a separate PR?

#1202

@mavam mavam changed the title Implement Model substitution Implement model substitution Nov 27, 2020
@mavam
Copy link
Member

mavam commented Nov 27, 2020

@tobim please rebase now that #1202 is merged.

Copy link
Member

@mavam mavam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot get this to work with the instructions provided in the PR. Here's what I tried:

vast start
vast import suricata < M57
vast import zeek < M57
vast export json 'net.connection == <_, _, _, _, _>'

I don't get any results, but was expecting to see all events that contain flow information.

libvast/vast/concept/parseable/vast/data.hpp Outdated Show resolved Hide resolved
libvast/vast/detail/vector_map.hpp Outdated Show resolved Hide resolved
libvast/src/taxonomies.cpp Outdated Show resolved Hide resolved
@mavam
Copy link
Member

mavam commented Nov 27, 2020

(It also looks like the YAML indentation is off.)

@tobim tobim force-pushed the tdhtf/ch1724/model-substitution branch from 085d6d4 to 79d26c4 Compare November 27, 2020 20:24
@tobim tobim requested a review from mavam November 30, 2020 13:06
Copy link
Member

@mavam mavam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me now. We are aware of the known deficiencies, and given that this is still marked as experimental features, we can merge this.

But it's nice queries like this now work, which I verified:

vast export json 'net.connection == <_, _, _, _, "icmp">'

Even though they still return a superset of the net.connection events, this is already really neat. Nice work!

CHANGELOG.md Outdated Show resolved Hide resolved
@tobim tobim force-pushed the tdhtf/ch1724/model-substitution branch from d65bf8a to 85e019b Compare December 1, 2020 07:48
@tobim tobim merged commit 185a513 into master Dec 1, 2020
@tobim tobim deleted the tdhtf/ch1724/model-substitution branch December 1, 2020 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
3 participants