Add notion of generic formatters/formatter factories #77

mstemm · 2021-08-25T22:24:19Z

Add the notion of "generic" event formatters and formatter factories
that can create a formatter for a given format string:

gen_filter.h defines two new classes: gen_event_formatter provides
the interface to format events:
- set_format(): set the output format and format string
- tostring(): resolve the format with info from a gen_event,
  populating a resolved string.
- tostring_withformat(): like tostring() but with a one-off output
  format.
- get_field_values(): return all templated field names and values
  from the configured format string.
- get_output_format(): get the current output format.
gen_event_formatter_factory performs a similar role as the existing
sinsp_evt_formatter_cache, in that it maintains a cache of previously
used format strings/formatters, to avoid the overhead of creating
formatters. It simply returns a formatter, though, rather than
duplicating the format methods like sinsp_evt_formatter_cache does.

This can be used in programs like falco to format general purpose
events without having a direct connection to an
inspector/filterchecks/etc.

The eventformatter changes simply add gen_event_formatter as a
parent class and implements the interfaces. To aid in backwards
compatibility with other parts of libsinsp, this only adds new
methods as needed to conform to the gen_event_formatter
interface. In some cases, the new methods just call existing methods
that did the same thing.

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area driver-kmod

/area driver-ebpf

/area libscap

/area libsinsp

/area tests

/area proposals

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

new(libsinsp): add notion of generic event formatters/formatter factories, similar to how filters have generic filters/factories

Modify falco_formats to only be responsible for resolving a rule's output string or coming up with a map of field name->field values from a given output string. It relies on the changes in falcosecurity/libs#77 to use generic formatters for a given source. Remove lua bindings to create a formatter/free a formatter. Those were unused as of the changes in #1451, so finally remove them now. Signed-off-by: Mark Stemm <mark.stemm@gmail.com>

gnosek

Can't say I like the duplicated APIs introduced but there probably is a reason for this

userspace/libsinsp/eventformatter.cpp

mstemm · 2021-09-10T21:19:53Z

/test build-libs

mstemm · 2021-09-10T21:20:08Z

/test build-libs-with-chisels

leogr

The duplicated API is not ideal, otherwise SGTM.
But I can't think of a way around that, so approved.
/approve

poiana · 2021-09-17T13:06:38Z

LGTM label has been added.

Git tree hash: e197fd830089ad2fa969a2fb4657bbf6b2326d2c

poiana · 2021-10-01T16:40:00Z

LGTM label has been added.

Git tree hash: 95ea41f503d52f247f1be655f357d469e0cee0e1

poiana · 2021-10-01T16:51:32Z

LGTM label has been added.

Git tree hash: b9c0994277c309231bae5d30f2b9df166dfa7c01

leogr

/approve

poiana · 2021-10-04T08:00:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leogr, mstemm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Add the notion of "generic" event formatters and formatter factories that can create a formatter for a given format string: - gen_filter.h defines two new classes: gen_event_formatter provides the interface to format events: - set_format(): set the output format and format string - tostring(): resolve the format with info from a gen_event, populating a resolved string. - tostring_withformat(): like tostring() but with a one-off output format. - get_field_values(): return all templated field names and values from the configured format string. - get_output_format(): get the current output format. - gen_event_formatter_factory performs a similar role as the existing sinsp_evt_formatter_cache, in that it maintains a cache of previously used format strings/formatters, to avoid the overhead of creating formatters. It simply returns a formatter, though, rather than duplicating the format methods like sinsp_evt_formatter_cache does. This can be used in programs like falco to format general purpose events without having a direct connection to an inspector/filterchecks/etc. - The eventformatter changes simply add gen_event_formatter as a parent class and implements the interfaces. To aid in backwards compatibility with other parts of libsinsp, this only adds new methods as needed to conform to the gen_event_formatter interface. In some cases, the new methods just call existing methods that did the same thing. Signed-off-by: Mark Stemm <mark.stemm@gmail.com>

poiana · 2021-10-04T15:17:25Z

LGTM label has been added.

Git tree hash: 5b20a78ca3875e164fcb5faeed84e1163579dbdf

Pinned to a commit that contains PRs falcosecurity/libs#74, falcosecurity/libs#75, falcosecurity/libs#76, falcosecurity/libs#77 Signed-off-by: Mark Stemm <mark.stemm@gmail.com>

Modify falco_formats to only be responsible for resolving a rule's output string or coming up with a map of field name->field values from a given output string. It relies on the changes in falcosecurity/libs#77 to use generic formatters for a given source. Remove lua bindings to create a formatter/free a formatter. Those were unused as of the changes in #1451, so finally remove them now. Signed-off-by: Mark Stemm <mark.stemm@gmail.com>

In #74 we pushed down some code from falco that determines the sinsp event types that are applicable for a given filter. The set of event types starts with "all events", and as the filter condition is parsed, including logical operators like "and", "or", "not", etc, the set of event types is changed, honoring the logical operators. For example, for a condition proc.name=nginx, the filter is applicable for all event types, as the condition does not include any evt.type field. For a more complicated condition like evt.type=openat and proc.name=nginx, the first field restricts the event types to openat, which is logical anded against all event types from proc.name, resulting in only the event type openat. With the introduction of plugins, there's also a need to segregate plugin-related filterchecks from non-plugin-related filterchecks by event source, but that's handled at a higher level, using a notion of filter factories and formatter factories (#77). The bug is that plugin filtercheck fields like ct.name, json.value were mistakenly being restricted to only the plugin event PPME_PLUGINEVENT_E. This was being mistakenly inverted when conditions had a "not" operator, with the result being that they did not run on any event types at all. The fix is to treat plugin fields as working with all event types, just like almost all other fields like proc.name, etc. are. This allows the logical operators to combine event type sets properly. This, along with other small changes in falco + plugins, fixes falcosecurity/plugins#56. Signed-off-by: Mark Stemm <mark.stemm@gmail.com>

poiana added release-note kind/feature New feature or request dco-signoff: no labels Aug 25, 2021

poiana requested review from gnosek and leogr August 25, 2021 22:24

poiana added area/libsinsp size/L labels Aug 25, 2021

mstemm force-pushed the general-purpose-formatters branch from 9e173ce to c301856 Compare August 26, 2021 01:26

poiana added dco-signoff: yes and removed dco-signoff: no labels Aug 26, 2021

mstemm mentioned this pull request Aug 27, 2021

Gen events falco engine falcosecurity/falco#1715

Merged

gnosek reviewed Sep 9, 2021

View reviewed changes

userspace/libsinsp/eventformatter.cpp Outdated Show resolved Hide resolved

userspace/libsinsp/eventformatter.cpp Outdated Show resolved Hide resolved

mstemm force-pushed the general-purpose-formatters branch from 2e19f82 to d7b7fb3 Compare September 10, 2021 19:59

mstemm mentioned this pull request Sep 16, 2021

Add ability to return all fields exported by a factory #75

Merged

leogr previously approved these changes Sep 17, 2021

View reviewed changes

poiana assigned leogr Sep 17, 2021

poiana added the lgtm label Sep 17, 2021

poiana added the approved label Sep 17, 2021

mstemm dismissed leogr’s stale review via 27e0ecb October 1, 2021 16:39

poiana added dco-signoff: no and removed dco-signoff: yes lgtm labels Oct 1, 2021

mstemm force-pushed the general-purpose-formatters branch from 27e0ecb to 3f1123e Compare October 1, 2021 16:39

poiana added dco-signoff: yes and removed dco-signoff: no labels Oct 1, 2021

gnosek previously approved these changes Oct 1, 2021

View reviewed changes

mstemm dismissed gnosek’s stale review via 6ae437b October 1, 2021 16:50

mstemm force-pushed the general-purpose-formatters branch from 3f1123e to 6ae437b Compare October 1, 2021 16:50

poiana removed the lgtm label Oct 1, 2021

gnosek previously approved these changes Oct 1, 2021

View reviewed changes

poiana added the lgtm label Oct 1, 2021

leogr previously approved these changes Oct 4, 2021

View reviewed changes

mstemm dismissed stale reviews from leogr and gnosek via 805e466 October 4, 2021 15:15

mstemm force-pushed the general-purpose-formatters branch from 6ae437b to 805e466 Compare October 4, 2021 15:15

poiana removed the lgtm label Oct 4, 2021

gnosek approved these changes Oct 4, 2021

View reviewed changes

poiana added the lgtm label Oct 4, 2021

leogr approved these changes Oct 4, 2021

View reviewed changes

poiana merged commit a5ea367 into master Oct 4, 2021

poiana deleted the general-purpose-formatters branch October 4, 2021 15:43

mstemm mentioned this pull request Jan 26, 2022

empty string with double quotes in aws_cloudtrail event "" falcosecurity/plugins#56

Closed

mstemm mentioned this pull request Jan 26, 2022

Treat plugin fields like other sinsp fields (all evt types) #190

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add notion of generic formatters/formatter factories #77

Add notion of generic formatters/formatter factories #77

mstemm commented Aug 25, 2021 •

edited by leogr

Loading

gnosek left a comment

mstemm commented Sep 10, 2021

mstemm commented Sep 10, 2021

leogr left a comment •

edited

Loading

poiana commented Sep 17, 2021

poiana commented Oct 1, 2021

poiana commented Oct 1, 2021

leogr left a comment

poiana commented Oct 4, 2021

poiana commented Oct 4, 2021

Add notion of generic formatters/formatter factories #77

Add notion of generic formatters/formatter factories #77

Conversation

mstemm commented Aug 25, 2021 • edited by leogr Loading

gnosek left a comment

Choose a reason for hiding this comment

mstemm commented Sep 10, 2021

mstemm commented Sep 10, 2021

leogr left a comment • edited Loading

Choose a reason for hiding this comment

poiana commented Sep 17, 2021

poiana commented Oct 1, 2021

poiana commented Oct 1, 2021

leogr left a comment

Choose a reason for hiding this comment

poiana commented Oct 4, 2021

poiana commented Oct 4, 2021

mstemm commented Aug 25, 2021 •

edited by leogr

Loading

leogr left a comment •

edited

Loading