Repository navigation
using osqueryd
osqueryd is the host monitoring daemon which is included with osquery.
osqueryd allows you to schedule queries for execution across your infrastructure. The daemon aggregates query results over time and generates logs which indicate state changes in your infrastructure. These logs can be used to gain insight into the security, performance, configuration and state of your infrastructure.
osqueryd can use your existing infrastructure because of it's plugin architecture.
osqueryd works by allowing you to specify a SQL query which represents interesting information about an operating system. If the result of a scheduled query changes, you should know about it. Consider a query like SELECT name FROM kextstat;. On OS X, this query will return the names of all currently loaded kernel extensions. Perhaps, in your environment, that's interesting information and you'd like to know whenever that changes (ie: a new kernel extension is added or one is removed). If you schedule that query, osqueryd will tell you whenever the result set changes. So by scheduling SELECT * FROM kextstat; with an interval of 10, osqueryd will let you know whenever a new kernel extension has been added or removed with a refresh interval of 10 seconds.
Scheduled queries are configured via the config interface. The default here is to just read a JSON file from the local filesystem, but more options exist. If you're interested in integrating osqueryd's config system with your environment, read the config options guide.
Regardless of what config retriever you use (again, the default is the "filesystem" retriever), config files will look kind of like this:
{
"scheduledQueries": [
{
"name": "awesome",
"query": "SELECT col1, col2, pid FROM awesome_table;",
"interval": 60
},
{
"name": "foobar",
"query": "SELECT foo, bar, pid FROM foobar_table;",
"interval": 600
}
]
}The above config tells osqueryd to schedule two queries: "awesome" and "foobar". Each query needs a unique name. This name is used internally to reference your query throughout the lifecycle of osqueryd. You then just need to include the text of the SQL query in the "query" field and how often you want it to execute in the "interval" field. So in the above example, "awesome" runs every minute and "foobar" runs every ten minutes.
The daemon takes care of taking each scheduled query and running the SQL at an interval that you define. The results of your query are cached on disk via RocksDB. The first run of the query will find no existing results in RocksDB, so it will log the entire result set. Every time that the query gets executed after the first execution, only changes will be logged.
You may distribute configurations in your environment differently than how Facebook distributes configurations. To support many environments, the way that osqueryd goes about retrieving configurations is completely pluggable and configurable. If you're interested in customizing the way that osquery acquires it's config, read the registering config plugins guide.
By default, running osqueryd --help will tell you the default locations of logs. This is traditionally /var/log/osquery/, but this is subject to change. There are two sets of logs that get stored in this directory:
- Application info, error and warning logs
- osqueryd scheduled query results logs
Consider the following directory listing:
$ ls -l /var/log/osquery/
total 24
lrwxr-xr-x 1 root wheel 77 Sep 30 17:37 osqueryd.INFO -> osqueryd.hostname.root.log.INFO.20140930-173730.97830
-rw-r--r-- 1 root wheel 1226 Sep 30 17:37 osqueryd.hostname.root.log.INFO.20140930-173730.97830
-rw-r--r-- 1 root wheel 388 Sep 30 17:37 osqueryd.results.log
Application logs are generated by the glog logging framework.
As the above directory listing indicates, "osqueryd.INFO" is a symlink to the most recent execution's INFO log. The same is true for the warning, error and fatal logs. For more information on the format of GLOG logs, please refer to the GLOG documentation.
The results of your scheduled queries are logged to the "results log". This log is formatted a bit differently to make it easier to do automatic processing of the data. Each line of the log is a complete JSON string which indicates what data has been added/removed by which query.
If a scheduled query indicates many individual state changes, the batched format will include all results in a single log line. If you're programmatically parsing lines and loading them into a backend datastore, this is probably the best solution.
To enable batch log lines, launch osqueryd with the --log_result_events=false argument.
Consider a query such as SELECT name, path, pid FROM processes;. Although each results log item is on a single line, for visual purposes, the following example is expanded across multiple lines:
{
"diffResults": {
"added": [
{
"name": "osqueryd",
"path": "/usr/local/bin/osqueryd",
"pid": "97830"
}
],
"removed": [
{
"name": "osqueryd",
"path": "/usr/local/bin/osqueryd",
"pid": "97650"
}
]
},
"name": "processes",
"hostname": "hostname.local",
"calendarTime": "Tue Sep 30 17:37:30 2014",
"unixTime": "1412123850"
}If you're directly consuming osqueryd logs into a log aggregation system like Logstash or Splunk, then it's easier if all of the events occur as a separate log line. This makes ad-hoc analytics on log lines much easier.
Consider a query such as SELECT name, path, pid FROM processes;. Although each results log item is on a single line, for visual purposes, the following examples are expanded across multiple lines:
{
"action": "added",
"columns": {
"name": "osqueryd",
"path": "/usr/local/bin/osqueryd",
"pid": "97830"
},
"name": "processes",
"hostname": "hostname.local",
"calendarTime": "Tue Sep 30 17:37:30 2014",
"unixTime": "1412123850"
}{
"action": "removed",
"columns": {
"name": "osqueryd",
"path": "/usr/local/bin/osqueryd",
"pid": "97650"
},
"name": "processes",
"hostname": "hostname.local",
"calendarTime": "Tue Sep 30 17:37:30 2014",
"unixTime": "1412123850"
}Notice how the log is broken up into two log lines in this example. This is the primary way that batch log events differ from event-based log events. The event-based log format is the default.
In both of the above examples, the log tells us that a binary called "osqueryd" was stopped and a new binary with the same name was started (note that they have different pids). The data is generated by keeping a cache of previous query results and only logging when the cache changes. If no new processes are started or stopped, this query won't log any results.
The key concept here is that each log line represents a state change in an operating system. The format of logs are the same for all scheduled queries, which allows you to easily add new queries which take advantage of existing infrastructure (logging, tracking, etc).
osqueryd's application logs are always logged to the filesystem. If you want to aggregate these logs, use your favorite traditional host log aggregation solution (rsyslog, logstash, etc).
osqueryd's results logs are designed to be much more configurable. If you'd like to control the way that osquery logs results with more granularity, read the registering logger plugins guide.