Addl docker container info #655

mstemm · 2016-09-08T23:24:21Z

Add new visibility into containers (specifically docker) with filterchecks that return whether or not a container is privileged and information on mounts made by a container.

Only implemented for docker containers right now.

I plan on using these to implement new falco rules that detect privileged containers and containers that mount sensitive host directories like '/proc', etc.

@luca3m @gianlucaborello I'd love feedback on the mount stuff. I tried to come up with something flexible and not too complex but I'd like to get your impressions on it.

Checking in separate from other changes.

Change glob matching to allow *, ? to match the path separator /. This allows for matching /proc/* as anything starting with /proc instead of having to match all path components.

New rule 'File Open by Privileged Container' triggers when a container that is running privileged opens a file. New rule 'Sensitive Mount by Container' triggers when a container that has a sensitive mount opens a file. Currently, a sensitive mount is a mount of /proc/*. This depends on draios/sysdig#655.

mstemm · 2016-09-09T00:16:11Z

Whoops, looks like I need to add dumping support--I'm noticing that the mounts/privileged state isn't being properly saved in .scap files. Let me fix that.

In sinsp_container_manager::parse_docker(), parse the Privileged field out of the json output and use it to set the instance variable m_privileged. Modify container_to_json/parse_container_json_evt to dump/read the privileged status from .scap files. New filtercheck container.privileged returns the privileged status as a boolean. Although other container types support the notion of privileged, I only support this filtercheck for docker containers. For non-docker containers, container.privileged returns NULL. Write privileged to/from scap files.

In sinsp_container_manager::parse_docker(), parse the Mounts field into a vector of class container_mount_info objects. Modify container_to_json/parse_container_json_evt to dump/read the mount information from .scap files. New filtercheck container.mounts displays the mount information as a comma-separated list of mount tuples <source>:<dest>:<mode>:<rdwr>:<propagation>, for example: mounts=/tmp:/foo/tmp::true::,/var/lib/docker/volumes/51c63c9efcd052551dd4898736dffb2692acbf6afd8d3f4d2f0cb89a7f8ace4a/_data:/data::true:rprivate: Note how empty values for mode/propagation turn into empty strings. This output is pretty awkward and verbose, so there are additional filterchecks that allow you to select any given attribute from a single mount. New filterchecks container.mount.{source,dest,dest,mode,rdwr,propagation} allow selecting any of the mount information indexed either by mount id "container.mount.mode[0]" or mount source "container.mount.mode[/var/log]". container.mount.source is different in that it's indexed by the mount destination instead of the mount source. When selecting a mount by source/dest pathname, you can provide a glob match instead (e.g. container.mount.mode[/var/log/*]). In this case, the first matching mount is returned. You can also use this indexing to return all the mount information for a single mount. New filtercheck container.mount returns the tuple information for a single mount, indexed by number (container.mount[0]) or source path/glob (container.mount[/var/log]. This can also be used to test if a mount exists, like container.mount[/proc/*] to detect containers that mount /proc or other sensitive directories. Write mounts to/from .scap files.

New rule 'File Open by Privileged Container' triggers when a container that is running privileged opens a file. New rule 'Sensitive Mount by Container' triggers when a container that has a sensitive mount opens a file. Currently, a sensitive mount is a mount of /proc. This depends on draios/sysdig#655.

mstemm · 2016-09-09T19:58:43Z

Ok, file support added. Wanna take a look?

luca3m · 2016-09-12T13:14:55Z

Looks good to me, nice addition 👍

gianlucaborello · 2016-09-12T17:42:31Z

userspace/libsinsp/container.h

 	string m_id;
 	sinsp_container_type m_type;
 	string m_name;
 	string m_image;
 	uint32_t m_container_ip;
+	bool m_privileged;


Are there cases where this member remains uninitialized and the user can read a spurious value?

For non-docker containers, the filtercheck always returns NULL. For docker containers, m_privileged should be set while parsing the results of the GET /containers/XXX/json response.I think the result always has a Privileged field.

Regardless, I should initialize it to false just to make sure. I've got that fix in #658.

I was more thinking of a case where you read an old trace file that contains a Docker json blob without the privileged flag and the customer maybe runs sysdig or falco and gets spurious privileged values on that trace, just wanted to make sure that was covered.

* Whitespace changes. Checking in separate from other changes. * Change glob matching to allow path separators. Change glob matching to allow *, ? to match the path separator /. This allows for matching /proc/* as anything starting with /proc instead of having to match all path components. * Add ability to detect/show privileged status. In sinsp_container_manager::parse_docker(), parse the Privileged field out of the json output and use it to set the instance variable m_privileged. Modify container_to_json/parse_container_json_evt to dump/read the privileged status from .scap files. New filtercheck container.privileged returns the privileged status as a boolean. Although other container types support the notion of privileged, I only support this filtercheck for docker containers. For non-docker containers, container.privileged returns NULL. Write privileged to/from scap files. * Add ability to display container mount info. In sinsp_container_manager::parse_docker(), parse the Mounts field into a vector of class container_mount_info objects. Modify container_to_json/parse_container_json_evt to dump/read the mount information from .scap files. New filtercheck container.mounts displays the mount information as a comma-separated list of mount tuples <source>:<dest>:<mode>:<rdwr>:<propagation>, for example: mounts=/tmp:/foo/tmp::true::,/var/lib/docker/volumes/51c63c9efcd052551dd4898736dffb2692acbf6afd8d3f4d2f0cb89a7f8ace4a/_data:/data::true:rprivate: Note how empty values for mode/propagation turn into empty strings. This output is pretty awkward and verbose, so there are additional filterchecks that allow you to select any given attribute from a single mount. New filterchecks container.mount.{source,dest,dest,mode,rdwr,propagation} allow selecting any of the mount information indexed either by mount id "container.mount.mode[0]" or mount source "container.mount.mode[/var/log]". container.mount.source is different in that it's indexed by the mount destination instead of the mount source. When selecting a mount by source/dest pathname, you can provide a glob match instead (e.g. container.mount.mode[/var/log/*]). In this case, the first matching mount is returned. You can also use this indexing to return all the mount information for a single mount. New filtercheck container.mount returns the tuple information for a single mount, indexed by number (container.mount[0]) or source path/glob (container.mount[/var/log]. This can also be used to test if a mount exists, like container.mount[/proc/*] to detect containers that mount /proc or other sensitive directories. Write mounts to/from .scap files.

New rule 'File Open by Privileged Container' triggers when a container that is running privileged opens a file. New rule 'Sensitive Mount by Container' triggers when a container that has a sensitive mount opens a file. Currently, a sensitive mount is a mount of /proc. This depends on draios/sysdig#655.

* Create embeddable falco engine. Create standalone classes falco_engine/falco_outputs that can be embedded in other programs. falco_engine is responsible for matching events against rules, and falco_output is responsible for formatting an alert string given an event and writing the alert string to all configured outputs. falco_engine's main interfaces are: - load_rules/load_rules_file: Given a path to a rules file or a string containing a set of rules, load the rules. Also loads needed lua code. - process_event(): check the event against the set of rules and return the results of a match, if any. - describe_rule(): print details on a specific rule or all rules. - print_stats(): print stats on the rules that matched. - enable_rule(): enable/disable any rules matching a pattern. New falco command line option -D allows you to disable one or more rules on the command line. falco_output's main interfaces are: - init(): load needed lua code. - add_output(): add an output channel for alert notifications. - handle_event(): given an event that matches one or more rules, format an alert message and send it to any output channels. Each of falco_engine/falco_output maintains a separate lua state and loads separate sets of lua files. The code to create and initialize the lua state is in a base class falco_common. falco_engine no longer logs anything. In the case of errors, it throws exceptions. falco_logger is now only used as a logging mechanism for falco itself and as an output method for alert messages. (This should really probably be split, but it's ok for now). falco_engine contains an sinsp_evttype_filter object containing the set of eventtype filters. Instead of calling m_inspector->add_evttype_filter() to add a filter created by the compiler, call falco_engine::add_evttype_filter() instead. This means that the inspector runs with a NULL filter and all events are returned from do_inspect. This depends on draios/sysdig#633 which has a wrapper around a set of eventtype filters. Some additional changes along with creating these classes: - Some cleanups of unnecessary header files, cmake include_directory()s, etc to only include necessary includes and only include them in header files when required. - Try to avoid 'using namespace std' in header files, or assuming someone else has done that. Generally add 'using namespace std' to all source files. - Instead of using sinsp_exception for all errors, define a falco_engine_exception class for exceptions coming from the falco engine and use it instead. For falco program code, switch to general exceptions under std::exception and catch + display an error for all exceptions, not just sinsp_exceptions. - Remove fields.{cpp,h}. This was dead code. - Start tracking counts of rules by priority string (i.e. what's in the falco rules file) as compared to priority level (i.e. roughtly corresponding to a syslog level). This keeps the rule processing and rule output halves separate. This led to some test changes. The regex used in the test is now case insensitive to be a bit more flexible. - Now that draios/sysdig#632 is merged, we can delete the rules object (and its lua_parser) safely. - Move loading the initial lua script to the constructor. Otherwise, calling load_rules() twice re-loads the lua script and throws away any state like the mapping from rule index to rule. - Allow an empty rules file. Finally, fix most memory leaks found by valgrind: - falco_configuration wasn't deleting the allocated m_config yaml config. - several ifstreams were being created simply to test which falco config file to use. - In the lua output methods, an event formatter was being created using falco.formatter() but there was no corresponding free_formatter(). This depends on changes in draios/sysdig#640. * Move falco engine to its own library. Move the c++ and lua code implementing falco engine/falco common to its own directory userspace/engine. It's compiled as a static library libfalco_engine.a, and has its own CMakeLists.txt so it can be included by other projects. The engine's CMakeLists.txt has a add_subdirectory for the falco rules directory, so including the engine also builds the rules. The variables you need to set to use the engine's CMakeLists.txt are: - CMAKE_INSTALL_PREFIX: the root directory below which everything is installed. - FALCO_ETC_DIR: where to install the rules file. - FALCO_SHARE_DIR: where to install lua code, relative to the - install/package root. - LUAJIT_INCLUDE: where to find header files for lua. - FALCO_SINSP_LIBRARY: the library containing sinsp code. It will be - considered a dependency of the engine. - LPEG_LIB/LYAML_LIB/LIBYAML_LIB: locations for third-party libraries. - FALCO_COMPONENT: if set, will be included as a part of any install() commands. Instead of specifying /usr/share/falco in config_falco_*.h.in, use CMAKE_INSTALL_PREFIX and FALCO_SHARE_DIR. The lua code for the engine has also moved, so the two lua source directories (userspace/engine/lua and userspace/falco/lua) need to be available separately via falco_common, so make it an argument to falco_common::init. As a part of making it easy to include in another project, also clean up LPEG build/defs. Modify build-lpeg to add a PREFIX argument to allow for object files/libraries being in an alternate location, and when building lpeg, put object files in a build/ subdirectory. * Add ignores for test-related files. Ignore results.json and similar names. Also ignore the file created when running phoronix tests. * Add configurable event dropping for falco engine. Add the ability to drop events at the falco engine level in a way that can scale with the dropping that already occurs at the kernel/inspector level. New inline function should_drop_evt() controls whether or not events are matched against the set of rules, and is controlled by two values--sampling ratio and sampling multiplier. Here's how the sampling ratio and multiplier influence whether or not an event is dropped in should_drop_evt(). The intent is that m_sampling_ratio is generally changing external to the engine e.g. in the main inspector class based on how busy the inspector is. A sampling ratio implies no dropping. Values > 1 imply increasing levels of dropping. External to the engine, the sampling ratio results in events being dropped at the kernel/inspector interface. The sampling multiplier is an amplification to the sampling factor in m_sampling_ratio. If 0, no additional events are dropped other than those that might be dropped by the kernel/inspector interface. If 1, events that make it past the kernel module are subject to an additional level of dropping at the falco engine, scaling with the sampling ratio in m_sampling_ratio. Unlike the dropping that occurs at the kernel level, where the events in the first part of each second are dropped, this dropping is random. * Add tests for multiple files, disabled rules. Add test that cover reading from multiple sets of rule files and disabling rules. Specific changes: - Modify falco to allow multiple -r arguments to read from multiple files. - In the test multiplex file, add a disabled_rules attribute, containing a sequence of rules to disable. Result in -D arguments when running falco. - In the test multiplex file, 'rules_file' can be a sequence. It results in multiple -r arguments when running falco. - In the test multiplex file, 'detect_level' can be a squence of multiple severity levels. All levels will be checked for in the output. - Move all test rules files to a rules subdirectory and all trace files to a traces subdirectory. - Add a small trace file for a simple cat of /dev/null. Used by the new tests. - Add the following new tests: - Reading from multiple files, with the first file being empty. Ensure that the rules from the second file are properly loaded. - Reading from multiple files with the last being empty. Ensures that the empty file doesn't overwrite anything from the first file. - Reading from multiple files with varying severity levels for each rule. Ensures that both files are properly read. - Disabling rules from a rules file, both with full rule names and regexes. Will result in not detecting anything. * Fix docker builds. gnupg2 is missing on latest debian:unstable. * Don't run the spawned program in a shell. Instead, run it directly. This avoids false positives when running non-bash commands and false negatives when trying to run a shell. * Eliminate FPs. Docker 1.12 split docker into docker and dockerd, so add dockerd as a docker binary. Also be consistent about using docker_binares instead of just references to docker. Also add ldconfig as a program that can write to files below /etc. * Handle dbus-daemon-launch-helper. It starts dbus-daemon. Process names are truncated, though, so use dbus-daemon-lau. * Program/docker image that performs bad activities. C++ program that performs bad activities related to the current falco ruleset. There are configurable actions for almost all of the current ruleset, via the --action argument. By default runs in a loop forever. Can be overridden via --once. Also add a Dockerfile that compiles event_generator.cpp within an alpine linux image and copies it to /usr/local/bin. This image has been pushed to docker hub as "sysdig/falco-event-generator:latest". Add a Makefile that runs the right docker build command. * Improve ruleset based on falco event-generator. Improve ruleset after using with falco event_generator: - Instead of assuming all shells are bash, add a list shell_binaries and macro shell_procs, and replace references to bash with shell_procs. This revealed some other programs that can spawn shells. - Add "login" as an interactive command. systemd-login isn't in alpine linux, which is the linux distro used for the container. - Move read_sensitive_file_untrusted before read_sensitive_file_trusted_after_startup, so it can hit first. * Install falco rules with configurable filename. New variable FALCO_RULES_DEST_FILENAME allows the rules file to be installed with a different filename. Not set in the falco repo, but in the agent repo it's installed as falco_rules.default.yaml. * Verifying rule names can have spaces. Related to discussion on draios/agent#160, verifying we can have rule names with spaces. * Change rule names to be human readable. Given the prior test, change all rule names to be human readable. This is especially important for the agent integration as they are visible. * Don't alert on falco program notifications. Falco itself spawns a shell when using program notifications, so add falco to the set of trusted programs. (Also add some other programs like make, awk, configure, that are run while building). * Fix output methods that take configurations. The falco engine changes broke the output methods that take configuration (like the filename for file output, or the program for program output). Fix that by properly passing the options argument to each method's output function. * Add regression tests for configurable outputs. - In the regression tests, make the config file configurable in the multiplex file via 'conf_file'. - A new multiplex file item 'outputs' containing a list of <filename>: <regex> tuples. For each item, the test reads the file and matches each line against the regex. A match must be found for the test to pass. - Add 2 new tests that test file output and program output. They write to files below /tmp/falco_outputs/ and the contents are checked to ensure that alerts are written. * Support enabled flag for rules. If a rule has a enabled attribute, and if the value is false, call the engine's enable_rule() method to disable the rule. Like add_filter, there's a static method which takes the object as the first argument and a non-static method that calls the engine. This fixes #72. * Add test for enabled flag. New test case disables a rule that would otherwise match. * Include condition in compilation errors. When a macro/rule condition can't be compiled, include the condition in the error message. * Parser changes to support new sysdig features Support "glob" as an operator and allow pathnames to be the index into bracketed selectors of fields. * New rules related to containers. New rule 'File Open by Privileged Container' triggers when a container that is running privileged opens a file. New rule 'Sensitive Mount by Container' triggers when a container that has a sensitive mount opens a file. Currently, a sensitive mount is a mount of /proc. This depends on draios/sysdig#655. * Reduce FPs related to Kubernetes. The new privileged falco rule was noisy when running kubernetes, which can run privileged. Add it to the trusted_containers list. Also eliminate a couple spurious warnings related to spawning shells in containers. * Add the new pmatch operator. Make changes to the lua-specific rule parser/compiler to handle the pmatch operator. * Fix lua stack leak. Need to pop the results of process_event so the stack doesn't grow without bound. * Install gcc-4.9 from Debian Jessie repositories As luca did for the agent, install gcc 4.9 from the debian jesse repository, as it has been removed from unstable. * Add license comments to all source code. Add comment blocks to all source code w/ our gpl copyright notice. * Add exfiltration action, env-specified actions. Add an exfiltration action that reads /etc/shadow and sends the contents to a arbitrary ip address and port via a udp datagram. Add the ability to specify actions via the environment instead of the command line. If actions are specified via the environment, they replace any actions specified on the command line. * Add jq to docker images. Add jq to the docker image containing falco. jq is very handy for transforming json, which comes into play if you want to post to slack (or other) webhooks. * Add notes on how to post to slack webhooks. Add comments for program_output that show how to post to a slack webhook and an alernate logging method--came up in one of the github issues. * Alphabetize command line options. There are a lot of command line options now, so sort them alphabetically in the usage and getopt handling to make them easier to find. Also rename -p <pidfile> to -P <pidfile>, thinking ahead to the next commit. * Add k8s/mesos/container info to rule outputs Copy handling of -pk/-pm/-pc/-k/-m arguments from sysdig. All of the relevant code was already in the inspector so that was easy. The information from k8s/mesos/containers is used in two ways: - In rule outputs, if the format string contains %container.info, that is replaced with the value from -pk/-pm/-pc, if one of those options was provided. If no option was provided, %container.info is replaced with a generic %container.name (id=%container.id) instead. - If the format string does not contain %container.info, and one of -pk/-pm/-pc was provided, that is added to the end of the formatting string. - If -p was specified with a general value (i.e. not kubernetes/mesos/container), the value is simply added to the end and any %container.info is replaced with the generic value. * Fix logic for detecting conf files. The logic for detecting if a file exists was backwards. It would treat a file as existing if it could *not* be opened. Reverse that logic so it works. This fixes #135. * Allow falco to spawn shells in containers. Falco is allowed to spawn shells in containers as a part of its program output method. * Add k8s binaries as trusted programs Add a new list k8s_binaries and allow those binaries to do things like setns/spawn shells. It's not the case that all of these binaries actually do these things, but keeping it as a single list makes management easier. * Add stats on events processed/dropped. Collect stats on the number of events processed and dropped. When run with -v, print these stats. This duplicates syddig behavior and can be useful when dianosing problems related to dropped events throwing off internal state tracking. * Add ability to write trace files. Bring over functionality from sysdig to write trace files. This is easy as all of the code to actually write the files is in the inspector. This just handles the -w option and arguments. This can be useful to write a trace file in parallel with live event monitoring so you can reproduce it later. * Rule fixes for dragent. Make sure falco doesn't detect the things draios-agent does as suspicious. It's possible that you might run open source falco alongside sysdig cloud. App checks spawned by sysdig cloud binaries might also change namespace, so also allow children of sysdigcloud binaries to call setns. * Updating for 0.4.0. CHANGELOG for release notes, README to update version. * Finishing merge.

* Whitespace changes. Checking in separate from other changes. * Change glob matching to allow path separators. Change glob matching to allow *, ? to match the path separator /. This allows for matching /proc/* as anything starting with /proc instead of having to match all path components. * Add ability to detect/show privileged status. In sinsp_container_manager::parse_docker(), parse the Privileged field out of the json output and use it to set the instance variable m_privileged. Modify container_to_json/parse_container_json_evt to dump/read the privileged status from .scap files. New filtercheck container.privileged returns the privileged status as a boolean. Although other container types support the notion of privileged, I only support this filtercheck for docker containers. For non-docker containers, container.privileged returns NULL. Write privileged to/from scap files. * Add ability to display container mount info. In sinsp_container_manager::parse_docker(), parse the Mounts field into a vector of class container_mount_info objects. Modify container_to_json/parse_container_json_evt to dump/read the mount information from .scap files. New filtercheck container.mounts displays the mount information as a comma-separated list of mount tuples <source>:<dest>:<mode>:<rdwr>:<propagation>, for example: mounts=/tmp:/foo/tmp::true::,/var/lib/docker/volumes/51c63c9efcd052551dd4898736dffb2692acbf6afd8d3f4d2f0cb89a7f8ace4a/_data:/data::true:rprivate: Note how empty values for mode/propagation turn into empty strings. This output is pretty awkward and verbose, so there are additional filterchecks that allow you to select any given attribute from a single mount. New filterchecks container.mount.{source,dest,dest,mode,rdwr,propagation} allow selecting any of the mount information indexed either by mount id "container.mount.mode[0]" or mount source "container.mount.mode[/var/log]". container.mount.source is different in that it's indexed by the mount destination instead of the mount source. When selecting a mount by source/dest pathname, you can provide a glob match instead (e.g. container.mount.mode[/var/log/*]). In this case, the first matching mount is returned. You can also use this indexing to return all the mount information for a single mount. New filtercheck container.mount returns the tuple information for a single mount, indexed by number (container.mount[0]) or source path/glob (container.mount[/var/log]. This can also be used to test if a mount exists, like container.mount[/proc/*] to detect containers that mount /proc or other sensitive directories. Write mounts to/from .scap files.

New rule 'File Open by Privileged Container' triggers when a container that is running privileged opens a file. New rule 'Sensitive Mount by Container' triggers when a container that has a sensitive mount opens a file. Currently, a sensitive mount is a mount of /proc. This depends on draios/sysdig#655.

mstemm added 2 commits September 8, 2016 09:17

Whitespace changes.

a78e1da

Checking in separate from other changes.

Change glob matching to allow path separators.

3f726f9

Change glob matching to allow *, ? to match the path separator /. This allows for matching /proc/* as anything starting with /proc instead of having to match all path components.

mstemm added 2 commits September 9, 2016 11:12

mstemm force-pushed the addl-docker-container-info branch from 1857952 to 593dd0b Compare September 9, 2016 18:15

mstemm mentioned this pull request Sep 9, 2016

Addl container rules falcosecurity/falco#120

Merged

mstemm merged commit de076df into dev Sep 12, 2016

mstemm deleted the addl-docker-container-info branch September 12, 2016 16:51

gianlucaborello reviewed Sep 12, 2016
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addl docker container info #655

Addl docker container info #655

mstemm commented Sep 8, 2016

mstemm commented Sep 9, 2016

mstemm commented Sep 9, 2016

luca3m commented Sep 12, 2016

gianlucaborello Sep 12, 2016

mstemm Sep 12, 2016

gianlucaborello Sep 12, 2016

Addl docker container info #655

Addl docker container info #655

Conversation

mstemm commented Sep 8, 2016

mstemm commented Sep 9, 2016

mstemm commented Sep 9, 2016

luca3m commented Sep 12, 2016

gianlucaborello Sep 12, 2016

Choose a reason for hiding this comment

mstemm Sep 12, 2016

Choose a reason for hiding this comment

gianlucaborello Sep 12, 2016

Choose a reason for hiding this comment