update: improve support and tests for live-capture event selection #2432

jasondellaluce · 2023-02-24T15:44:01Z

What type of PR is this?

/kind bug

/kind design

Any specific area of the project related to this PR?

/area engine

/area tests

What this PR does / why we need it:

This follows up the contributions of #2428 by fixing a design issue, adapting the codebase to the latest updates in libsinsp, and by growing the unit tests for the part of code responsible for event selection during live captures.

~400 lines of code of this PR are for tests.

Which issue(s) this PR fixes:

Special notes for your reviewer:

A lot changed this week, so here's quick recap for how Falco deals with events:

When reading capture files, Falco will receive and match rules for all events in the files
When opening a live capture for an event source that is not syscall, Falco will behave as usual
When opening a live event capture for the syscall event source:
1. Falco will determine a base set of events to collect. This can be the default minimum set that ensures state consistency and stability in libsinsp, or a custom set of events defined by the user through the base_syscalls configuration field.
2. The final selected set of events is the union of the base set computed in the previous step, and the events matched by the loaded rules. This can now include generic events too (never supported up until now)
3. If the -A flag is not set, performance-heavy syscalls are forcefully removed from the selected set (such as IO-intensive syscalls)
4. Falco will configure the chosen kernel driver with the selected set of events, and will only receive events matching the set

What was missing? The current implementation incorrectly uses the ppm_event_code enumerative as single source of truth for defining the set of selected events. This has some faults, the first being that there can be information loss about generic events. Instead, Falco needs to operate with both ppm_sc_code and ppm_event_code for their different purposes:

ppm_event_code is required for correctly matching events in the rulesets at runtime. In fact, Falco will receive events of all kinds (internal, container, plugin, old versions from capture files...), and needs to recognize and index them properly.
ppm_sc_code is required to define the set of selected events to configure the kernel drivers with. ppm_sc_code is a richer enumerative that has no information loss about generic events and includes syscalls-related events only. The unit test suite now checks that everything works as expected.

What's still missing? The work and design discussion around falcosecurity/libs#889 will lead to fixing two remaining problems:

Information loss when converting a ppm_sc_code to a ppm_event_code and viceversa.
Have a unique representation for events related to tracepoints (e.g. switch, procexit) and all the rest of the syscall events.

None of this is blocking for Falco in its current feature development status. Point 1) has no impact because Falco does not rely on the libsinsp conversion utilities, but searches for ppm_sc_codes in the ruleset right on instead. Point 2) will require a minor refactor in Falco because we currently have tracepoint in a different selection sets, however this will only be non-functional changes.

Does this PR introduce a user-facing change?:

update: improve support and tests for live-capture event selection

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…sc and new engine features Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

The test now take in accoint pre/post-conditions of the actions, usage of the -A option, and the newly-introduced base_syscall user configuration. This also makes sure that the event selection properly handles generic events and options/configs precedence. Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…e103d57f97d74dd374f Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

jasondellaluce · 2023-02-24T15:45:44Z

/milestone 0.35.0

userspace/falco/app/actions/configure_interesting_sets.cpp

Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

incertum

@jasondellaluce nice work! Given master is currently a tad flawed, would vote for merging rather faster and then polishing in a follow up PR if needed. WDYT?

incertum · 2023-02-24T16:41:49Z

userspace/falco/app/actions/configure_interesting_sets.cpp

+
+	// selected events are the union of the rules events set and the
+	// base events set (either the default or the user-defined one)
+	s.selected_sc_set = rules_sc_set.merge(base_sc_set);


I like how now that we reverted to starting from sc_set we could simplify and clean up "base_syscalls" right away and be more consistent 🎉

userspace/falco/app/actions/configure_interesting_sets.cpp

incertum · 2023-02-26T21:41:44Z

userspace/engine/evttype_index_ruleset.cpp

@@ -163,11 +170,15 @@ void evttype_index_ruleset::add(
 		wrap->filter = filter;
 		if(rule.source == falco_common::syscall_source)
 		{
-			wrap->evttypes = libsinsp::filter::ast::ppm_event_codes(condition.get());
+			wrap->sc_codes = libsinsp::filter::ast::ppm_sc_codes(condition.get());


Please see some more libs fixes in falcosecurity/libs@338bff0 that aim to make sure we handle the tough corner cases and special snow flakes. This is probably a perfect reflection of why we were going back and forth between first resolving the evt.type string names to event codes vs syscall codes ... meaning we need a combination of the two approaches after all.

Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

incertum · 2023-03-01T07:22:48Z

Just noticed that selected_event_set disappeared completely? Wouldn't we still need to activate all crucial non syscalls events for userspace? In addition, we could consider removing the static sinsp state enforcement from libsinsp::events::sinsp_state_event_set() again since we pivoted in the approach? https://github.com/falcosecurity/libs/blob/50aa2a0c9bb5f7e3c61dc0f2470f513521e34ec2/userspace/libsinsp/events/sinsp_events.cpp#L174

jasondellaluce · 2023-03-01T08:00:49Z

Just noticed that selected_event_set disappeared completely? Wouldn't we still need to activate all crucial non syscalls events for userspace?

This event selection process only concerns events that could be ignored at kernel-level, and for now we have no way to express the selection of non-syscall events (e.g. container) . This means that we'll keep receiving all of them even after these changes. Most of non-syscall events are meta-events or tracepoint-based events. The former are generated internally by libsinsp as part of its internal machinery, and the latter can be selected in the separate selected_tp_set.

incertum · 2023-03-01T19:49:13Z

Just noticed that selected_event_set disappeared completely? Wouldn't we still need to activate all crucial non syscalls events for userspace?

This event selection process only concerns events that could be ignored at kernel-level, and for now we have no way to express the selection of non-syscall events (e.g. container) . This means that we'll keep receiving all of them even after these changes. Most of non-syscall events are meta-events or tracepoint-based events. The former are generated internally by libsinsp as part of its internal machinery, and the latter can be selected in the separate selected_tp_set.

Thank you Jason. Fair to say it was never needed? https://github.com/falcosecurity/falco/blob/0.34.1/userspace/falco/app_actions/configure_interesting_sets.cpp#L38

jasondellaluce · 2023-03-06T10:21:01Z

Thank you Jason. Fair to say it was never needed? https://github.com/falcosecurity/falco/blob/0.34.1/userspace/falco/app_actions/configure_interesting_sets.cpp#L38

At the point in time of 0.34.0/.1, it was needed for printing the output of -i and the warning messages related to -A with its old behavior:

falco/userspace/falco/app_actions/print_syscall_events.cpp

Line 76 in 9a4205c

const auto events = get_event_entries(false, m_state->ppm_event_info_of_interest);
https://github.com/falcosecurity/falco/blob/0.34.1/userspace/falco/app_actions/print_ignored_events.cpp#L55

At the current time, storing this in the app state does not have value anymore, as the event set is not used only by the Falco engine.

jasondellaluce · 2023-03-06T15:30:20Z

cc @falcosecurity/falco-maintainers for visibility.

…75375154a58 Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

incertum · 2023-03-06T20:16:03Z

Thanks @jasondellaluce, also for expanding the unit tests!

incertum · 2023-03-07T21:52:54Z

@jasondellaluce as we bump libs one more suggesting to update falco engine version as we will have new fields available as well, ty!

…7badbbd43e Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

…9960ac38c6 Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

jasondellaluce · 2023-03-08T16:40:12Z

Note, the Code scanning results / CodeQL failing job is currently failing just reporting some issues with the latest version of TBB recently updated in falcosecurity/libs.

…5e53053412 Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

FedeDP

/approve

poiana · 2023-03-09T08:12:39Z

LGTM label has been added.

Git tree hash: 920c332289d1cb0bfeeb389368eff1e51bf213d9

Andreagit97

/approve

poiana · 2023-03-09T08:38:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Andreagit97, FedeDP, jasondellaluce

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Andreagit97,FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jasondellaluce and others added 7 commits February 24, 2023 14:43

update(userspace/engine): support searching ppm_sc events in rulesets

97e3efd

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

refactor(userspace/falco): adapt event set selection to only use ppm_…

6932687

…sc and new engine features Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

chore(userspace/falco): rename source file using its action name

389c818

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

chore(unit_tests): move existing test in right directory

a8b3aac

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(cmake): bump libs and driver versions to f0468f32d8d730d0e70d8…

548785f

…e103d57f97d74dd374f Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

chore: fix typos

425e9dc

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

poiana added release-note kind/bug dco-signoff: yes kind/design area/engine area/tests approved labels Feb 24, 2023

poiana requested review from Kaizhe and leogr February 24, 2023 15:44

poiana added the size/XXL label Feb 24, 2023

poiana added this to the 0.35.0 milestone Feb 24, 2023

incertum reviewed Feb 24, 2023

View reviewed changes

userspace/falco/app/actions/configure_interesting_sets.cpp Outdated Show resolved Hide resolved

chore(userspace/falco): apply review suggestions

3877525

Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

incertum reviewed Feb 24, 2023

View reviewed changes

incertum reviewed Feb 26, 2023

View reviewed changes

incertum mentioned this pull request Feb 28, 2023

[New Feature]: User flag for base_syscalls.repair as extension to base_syscalls config overriding default sinsp state enforcement #2433

Closed

chore(userspace/falco): apply review suggestion

5c6bb23

Co-authored-by: Melissa Kilby <melissa.kilby.oss@gmail.com> Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

incertum mentioned this pull request Feb 28, 2023

cleanup(libsinsp): Extension of ppm sc API for corner cases (e.g. event_set_to_names and names_to_sc_set) falcosecurity/libs#915

Merged

jasondellaluce mentioned this pull request Mar 6, 2023

docs: added documentation about -A option falcosecurity/falco-website#840

Merged

jasondellaluce added 2 commits March 6, 2023 15:31

update(cmake): bump libs and drivers to c2e227697b2a410eeca47fef260f5…

a67472c

…75375154a58 Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

fix(unit_tests): adapt to connect4 corner cases

3132dcb

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

jasondellaluce added 3 commits March 8, 2023 11:28

update(cmake): bump libs and driver to 172043999fbf3d04c2c40f98cc7795…

4fe22e0

…7badbbd43e Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(userspace/engine): bump engine version to 17

6b8e23d

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(cmake): bump libs and driver to a513c696b68676b872c2ea59636e65…

9d8f248

…9960ac38c6 Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

update(cmake): bump libs and driver to 652d6d134d5c2b355467de5be92213…

a1a7c4f

…5e53053412 Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>

FedeDP approved these changes Mar 9, 2023

View reviewed changes

poiana assigned FedeDP Mar 9, 2023

poiana added the lgtm label Mar 9, 2023

Andreagit97 approved these changes Mar 9, 2023

View reviewed changes

poiana assigned Andreagit97 Mar 9, 2023

poiana merged commit 85729f3 into master Mar 9, 2023

poiana deleted the fix/evt-codes-sc-rulesets branch March 9, 2023 08:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update: improve support and tests for live-capture event selection #2432

update: improve support and tests for live-capture event selection #2432

jasondellaluce commented Feb 24, 2023 •

edited

Loading

jasondellaluce commented Feb 24, 2023

incertum left a comment

incertum Feb 24, 2023

incertum Feb 26, 2023

incertum commented Mar 1, 2023

jasondellaluce commented Mar 1, 2023

incertum commented Mar 1, 2023

jasondellaluce commented Mar 6, 2023

jasondellaluce commented Mar 6, 2023

incertum commented Mar 6, 2023

incertum commented Mar 7, 2023

jasondellaluce commented Mar 8, 2023

FedeDP left a comment

poiana commented Mar 9, 2023

Andreagit97 left a comment

poiana commented Mar 9, 2023

update: improve support and tests for live-capture event selection #2432

update: improve support and tests for live-capture event selection #2432

Conversation

jasondellaluce commented Feb 24, 2023 • edited Loading

What type of PR is this?

Any specific area of the project related to this PR?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

jasondellaluce commented Feb 24, 2023

incertum left a comment

Choose a reason for hiding this comment

incertum Feb 24, 2023

Choose a reason for hiding this comment

incertum Feb 26, 2023

Choose a reason for hiding this comment

incertum commented Mar 1, 2023

jasondellaluce commented Mar 1, 2023

incertum commented Mar 1, 2023

jasondellaluce commented Mar 6, 2023

jasondellaluce commented Mar 6, 2023

incertum commented Mar 6, 2023

incertum commented Mar 7, 2023

jasondellaluce commented Mar 8, 2023

FedeDP left a comment

Choose a reason for hiding this comment

poiana commented Mar 9, 2023

Andreagit97 left a comment

Choose a reason for hiding this comment

poiana commented Mar 9, 2023

jasondellaluce commented Feb 24, 2023 •

edited

Loading