Repository navigation
using osqueryd
osqueryd is the host monitoring daemon which is included with osquery. Running osqueryd --help will surface some interesting configurable options. As of October 1, 2014, the configurable options are as follows:
$ osqueryd --help
osqueryd
--help, -h Print help and usage information
--config_retriever The config plugin to use (ex: filesystem, http)
Default: filesystem
--config_path If using the filesystem config plugin, the path where
your osquery JSON config file can be found
Default: /var/osquery/osquery.conf
--log_receiver The logger plugin to use (ex: filesystem, scribe)
Default: filesystem
--log_dir The directory which you would like to store your
output logs
Default: /var/log/osquery/
-v Increase output verbosity
Example: -v=3
For detailed information on how you should configure options like log_dir and log_receiver, check out the logging options guide.
For detailed information on how you should configure options like config_path and config_retriever, check out the config options guide.
osqueryd is a low-footprint distributed host monitoring daemon which allows you to schedule queries to be executed across your entire infrastructure. The daemon takes care of aggregating the query results over time and generates logs which indicate state changes in your infrastructure. You can use this to maintain insight into the security, performance, configuration and state of your entire infrastructure. osqueryd's logging can integrate right into your internal log aggregation pipeline, regardless of your technology stack, via a robust plugin architecture.
osqueryd works by allowing you to specify a SQL query which represents interesting information about an operating system. If the result of a scheduled query changes, you should know about it. Consider a query like SELECT name FROM kextstat;. On OS X, this query will return the names of all currently loaded kernel extensions. Perhaps, in your environment, that's interesting information and you'd like to know whenever that changes (ie: a new kernel extension is added or one is removed). If you schedule that query, osqueryd will tell you whenever the result set changes. So by scheduling SELECT * FROM kextstat; with an interval of 10, osqueryd will let you know whenever a new kernel extension has been added or removed with a refresh interval of 10 seconds.
Scheduled queries are configured via the config interface. The default here is to just read a JSON file from the local filesystem, but more options exist. If you're interested in integrating osqueryd's config system with your environment, read the config options guide.
Regardless of what config retriever you use (again, the default is the "filesystem" retriever), config files will look kind of like this:
{
"scheduledQueries": [
{
"name": "awesome",
"query": "SELECT col1, col2, pid FROM awesome_table;",
"interval": 60
},
{
"name": "foobar",
"query": "SELECT foo, bar, pid FROM foobar_table;",
"interval": 600
}
]
}The above config tells osqueryd to schedule two queries: "awesome" and "foobar". Each query needs a unique name. This name is used internally to reference your query throughout the lifecycle of osqueryd. You then just need to include the text of the SQL query in the "query" field and how often you want it to execute in the "interval" field. So in the above example, "awesome" runs every minute and "foobar" runs every ten minutes.
The daemon takes care of taking each scheduled query and running the SQL at an interval that you define. The results of your query are cached on disk via RocksDB. The first run of the query will find no existing results in RocksDB, so it will log the entire result set. Every time that the query gets executed after the first execution, only changes will be logged.
You may distribute configurations in your environment differently than how Facebook distributes configurations. To support many environments, the way that osqueryd goes about retrieving configurations is completely pluggable and configurable.
By default, osqueryd will look for a JSON file on disk. If you distribute configurations via something like Zookeeper, etcd, etc. then you simply need to write a single function in C++ which somehow acquires a string of JSON.
Consider the following example, which retrieves config data from some arbitrary internal API:
#include "osquery/config/plugin.h"
#include "internal/config.h"
namespace osquery {
class InternalConfigPlugin : public ConfigPlugin {
public:
std::pair<osquery::Status, std::string> genConfig() {
internal::ConfigeApi api;
std::string content;
api.getConfig("osquery/osquery", &content);
return std::make_pair(Status(0, "OK"), content);
}
virtual ~InternalConfigPlugin() {}
};
REGISTER_CONFIG_PLUGIN("internal", std::make_shared<osquery::InternalConfigPlugin>());
}That's it. You're just implementing genConfig. Once you add a config plugin, it's automatically built into the available options for --config_retriever. This is done by the call to REGISTER_CONFIG_PLUGIN. Registering a config plugin is all you need to do to get your config plugin working. No need to edit any source files other that your own!
By default, running osqueryd --help will tell you the default locations of logs. This is traditionally /var/log/osquery/, but this is subject to change. There are two sets of logs that get stored in this directory:
- Application info, error and warning logs
- osqueryd scheduled query results logs
Consider the following directory listing:
$ ls -l /var/log/osquery/
total 24
lrwxr-xr-x 1 root wheel 77 Sep 30 17:37 osqueryd.INFO -> osqueryd.hostname.root.log.INFO.20140930-173730.97830
-rw-r--r-- 1 root wheel 1226 Sep 30 17:37 osqueryd.hostname.root.log.INFO.20140930-173730.97830
-rw-r--r-- 1 root wheel 388 Sep 30 17:37 osqueryd.results.log
Application logs are generated by the glog logging framework.
As the above directory listing indicates, "osqueryd.INFO" is a symlink to the most recent execution's INFO log. The same is true for the warning, error and fatal logs. For more information on the format of GLOG logs, please refer to the GLOG documentation.
The results of your scheduled queries are logged to the "results log". This log is formatted a bit differently to make it easier to do automatic processing of the data. Each line of the log is a complete JSON string which indicates what data has been added/removed by which query.
Consider a query such as SELECT name, path, pid FROM processes;. Although each results log item is on a single line, for visual purposes, the following example is expanded across multiple lines:
{
"diffResults": {
"added": [
{
"name": "osqueryd",
"path": "/usr/local/bin/osqueryd",
"pid": "97830"
}
],
"removed": [
{
"name": "osqueryd",
"path": "/usr/local/bin/osqueryd",
"pid": "97650"
}
]
},
"name": "processes",
"hostname": "hostname.local",
"calendarTime": "Tue Sep 30 17:37:30 2014",
"unixTime": "1412123850"
}In the above example, the log tells us that a binary called "osqueryd" was stopped and a new binary with the same name was started (note that they have different pids). The data is generated by keeping a cache of previous query results and only logging when the cache changes. If no new processes are started or stopped, this query won't log any results.
The key concept here is that each log line represents a state change in an operating system. The format of logs are the same for all scheduled queries, which allows you to easily add new queries which take advantage of existing infrastructure (logging, tracking, etc).
osqueryd's application logs are always logged to the filesystem. If you want to aggregate these logs, use your favorite traditional host log aggregation solution (rsyslog, logstash, etc).
osqueryd's results logs are designed to be much more configurable. If you'd like to control the way that osquery logs results with more granularity, read the registering logger plugins guide.