Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flag excess drops #561

Merged
merged 6 commits into from Mar 27, 2019

Conversation

Projects
None yet
3 participants
@mstemm
Copy link
Contributor

commented Mar 13, 2019

For more information on this PR, read our blog post on CVE-2019-8339.

Changes to monitor for dropped system calls/kernel buffer exhaustion and take configurable actions when this occurs.

Certainly related to and might be a fix for #545.

@mstemm mstemm requested review from mattpag and lorenzo-david Mar 13, 2019

Show resolved Hide resolved falco.yaml Outdated
Show resolved Hide resolved userspace/falco/overflow.cpp Outdated
@mattpag
Copy link
Contributor

left a comment

LGTM, just a couple of comments

Show resolved Hide resolved userspace/falco/overflow.cpp Outdated
Show resolved Hide resolved falco.yaml Outdated

@mstemm mstemm force-pushed the flag-excess-drops branch from 226b0e3 to 132034b Mar 26, 2019

mstemm added some commits Mar 11, 2019

Make stats file interval configurable
New argument --stats_interval=<msec> controls the interval at which
statistics are written to the stats file. The default is 5000 ms (5 sec)
which matches the prior hardcoded interval.

The stats interval is triggered via signals, so an interval below ~250ms
will probably interfere with falco's behavior.
Add ability to emit general purpose messages
A new method falco_outputs::handle_msg allows emitting generic messages
that have a "rule", message, and output fields, but aren't exactly tied
to any event and aren't passed through an event formatter.

This allows falco to emit "events" based on internal checks like kernel
buffer overflow detection.
Clean up newline handling for logging
Log messages from falco_logger::log may or may not have trailing
newlines. Handle both by always adding a newline to stderr logs and
always removing any newline from syslog logs.
Add method to get sequence from subkey
New variant of get_sequence that allows fetching a list of items from a
key + subkey, for example:

key:
  subkey:
    - list
    - items
    - here

Both use a shared method get_sequence_from_node().
Monitor syscall event drops + optional actions
Start actively monitoring the kernel buffer for syscall event drops,
which are visible in scap_stats.n_drops, and add the ability
to take actions when events are dropped. The -v (verbose) and
-s (stats filename) arguments also print out information on dropped
events, but they were only printed/logged without any actions.

In falco config you can specify one or more of the following actions to
take when falco notes system call drops:

 - ignore (do nothing)
 - log a critical message
 - emit an "internal" falco alert. It looks like any other alert with a
   time, "rule", message, and output fields but is not related to any
   rule in falco_rules.yaml/other rules files.
 - exit falco (the idea being that the restart would be monitored
   elsewhere).

A new module syscall_event_drop_mgr is called for every event and
collects scap stats every second. If in the prior second there were
drops, perform_actions() handles the actions.

To prevent potential flooding in high drop rate environments, actions
are goverened by a token bucket with a rate of 1 actions per 30 seconds,
with a max burst of 10 seconds. We might tune this later based on
experience in busy environments.

This might be considered a fix for
#545. It doesn't
specifically flag falco rules alerts when there are drops, but does
make it easier to notice when there are drops.
Add unit test for syscall event drop detection
Add unit tests for syscall event drop detection. First, add an optional
config option that artifically increments the drop count every
second. (This is only used for testing).

Then add test cases for each of the following:
 - No dropped events: should not see any log messages or alerts.
 - ignore action: should note the drops but not log messages or alert.
 - log action: should only see log messages for the dropped events.
 - alert action: should only see alerts for the dropped events.
 - exit action: should see log message noting the dropped event and exit
   with rc=1

A new trace file ping_sendto.scap has 10 seconds worth of events to
allow the periodic tracking of drops to kick in.

@mstemm mstemm force-pushed the flag-excess-drops branch from e8a0456 to dd45d44 Mar 26, 2019

@mstemm

This comment has been minimized.

Copy link
Contributor Author

commented Mar 27, 2019

Also added a wiki page documenting the feature: https://github.com/falcosecurity/falco/wiki/Actions-For-Dropped-System-Call-Events and notes to the falco config wiki page for the new config options.

@mstemm mstemm merged commit bdda640 into dev Mar 27, 2019

2 checks passed

Travis CI - Branch Build Passed
Details
Travis CI - Pull Request Build Passed
Details

@mstemm mstemm deleted the flag-excess-drops branch Apr 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.