Repository navigation
performance tooling
Performance is a core feature of osquery's visibility capability. However, the tool is very powerful and there are opportunities to ruin the performance guarantees with ill-formed queries.
This guide provides an overview and tutorial for assuring performance of the osquery scheduled queries, as well as performance-centric development practices/enforcements.
The osquery tooling provides a full-featured profiling script. The script can evaluate table, query, and scheduled query performance on a system. Before scheduling a set of queries on your enterprise hosts, it is best practice to measure the expected performance impact:
Consider the following osquery.conf:
{
"scheduledQueries": [
{
"name": "alf_services",
"query": "select service, process from alf_services where state != 0;",
"interval": 60
},
{
"name": "installed_applications",
"query": "select name, path, bundle_version, minimum_system_version, applescript_enabled, bundle_executable from apps;",
"interval": 60
},
{
"name": "all_kexts",
"query": "select name, version from kextstat;",
"interval": 60
},
{
"name": "non_apple_kexts",
"query": "select * from kextstat where name not like 'com.apple.%' and name != '__kernel__';",
"interval": 60
},
{
"name": "processes_binding_to_ports",
"query": "select distinct process.name, listening.port, listening.protocol, listening.family, listening.address, process.pid, proce$
"interval": 60
},
{
"name": "processes_not_on_disk",
"query": "select * from processes where on_disk != 1;",
"interval": 60
}
]
}Each query provides useful information and will run every minute. But what sort of impact will this have on the client machines?
For this we can use ./tools/profile.py to profile the queries by running them for a configured number of rounds and reporting the pre-defined performance category of each. A higher category result means higher impact. High impact queries should be avoided, but if the information is valuable consider running them less-often.
$ sudo -E python ./tools/profile.py --count 1 --rounds 4 --config osquery.conf
Profiling query: select * from kextstat where name not like 'com.apple.%' and name != '__kernel__';
non_apple_kexts (1/4): duration: 0 (0.513329029083) cpu_time: 0 (0.12185088) memory: 0 (6787072) fds: 0 (4) utilization: 1 (12.0)
non_apple_kexts (2/4): duration: 0 (0.507598876953) cpu_time: 0 (0.11338522) memory: 0 (6807552) fds: 0 (4) utilization: 1 (11.15)
non_apple_kexts (3/4): duration: 0 (0.514081001282) cpu_time: 0 (0.160709296) memory: 0 (6791168) fds: 0 (4) utilization: 1 (15.45)
non_apple_kexts (4/4): duration: 0 (0.508167028427) cpu_time: 0 (0.151464072) memory: 0 (6832128) fds: 0 (4) utilization: 1 (14.95)
non_apple_kexts avg: duration: 0 (0.510793983936) cpu_time: 0 (0.136852367) memory: 0 (6804480.0) fds: 0 (4.0) utilization: 1 (13.3875)
Profiling query: select name, path, bundle_version, minimum_system_version, applescript_enabled, bundle_executable from apps;
installed_applications (1/4): duration: 2 (1.01154088974) cpu_time: 1 (0.679006946) memory: 1 (9658368) fds: 0 (4) utilization: 2 (45.0666666667)
installed_applications (2/4): duration: 2 (1.00866603851) cpu_time: 1 (0.788424656) memory: 1 (9363456) fds: 0 (4) utilization: 3 (52.3)
installed_applications (3/4): duration: 2 (1.01094698906) cpu_time: 1 (0.571719704) memory: 1 (9289728) fds: 0 (4) utilization: 2 (37.8666666667)
installed_applications (4/4): duration: 2 (1.00963997841) cpu_time: 1 (0.531233553) memory: 1 (9027584) fds: 0 (4) utilization: 2 (35.1666666667)
installed_applications avg: duration: 2 (1.01019847393) cpu_time: 1 (0.64259621475) memory: 1 (9334784.0) fds: 0 (4.0) utilization: 2 (42.6)
Profiling query: select service, process from alf_services where state != 0;
alf_services (1/4): duration: 0 (0.507262945175) cpu_time: 0 (0.020714005) memory: 0 (5906432) fds: 0 (4) utilization: 0 (1.9)
alf_services (2/4): duration: 0 (0.50815486908) cpu_time: 0 (0.021529091) memory: 0 (5943296) fds: 0 (4) utilization: 0 (1.95)
alf_services (3/4): duration: 0 (0.508809089661) cpu_time: 0 (0.023757138) memory: 0 (5902336) fds: 0 (4) utilization: 0 (2.2)
alf_services (4/4): duration: 0 (0.508450984955) cpu_time: 0 (0.022736648) memory: 0 (5906432) fds: 0 (4) utilization: 0 (2.1)
alf_services avg: duration: 0 (0.508169472218) cpu_time: 0 (0.0221842205) memory: 0 (5914624.0) fds: 0 (4.0) utilization: 0 (2.0375)
Profiling query: select * from processes where on_disk != 1;
processes_not_on_disk (1/4): duration: 0 (0.512609958649) cpu_time: 0 (0.041570172) memory: 0 (5959680) fds: 0 (4) utilization: 0 (3.65)
processes_not_on_disk (2/4): duration: 0 (0.507596969604) cpu_time: 0 (0.03292302) memory: 0 (5984256) fds: 0 (4) utilization: 0 (3.15)
processes_not_on_disk (3/4): duration: 0 (0.506883144379) cpu_time: 0 (0.023162205) memory: 0 (5992448) fds: 0 (4) utilization: 0 (2.2)
processes_not_on_disk (4/4): duration: 0 (0.509166002274) cpu_time: 0 (0.026498948) memory: 0 (5984256) fds: 0 (4) utilization: 0 (2.5)
processes_not_on_disk avg: duration: 0 (0.509064018726) cpu_time: 0 (0.03103858625) memory: 0 (5980160.0) fds: 0 (4.0) utilization: 0 (2.875)
Profiling query: select name, version from kextstat;
all_kexts (1/4): duration: 0 (0.505537033081) cpu_time: 0 (0.098467126) memory: 0 (6692864) fds: 0 (4) utilization: 1 (9.75)
all_kexts (2/4): duration: 0 (0.505311012268) cpu_time: 0 (0.103915444) memory: 0 (6811648) fds: 0 (4) utilization: 1 (10.25)
all_kexts (3/4): duration: 0 (0.506618022919) cpu_time: 0 (0.106134466) memory: 0 (6701056) fds: 0 (4) utilization: 1 (10.45)
all_kexts (4/4): duration: 0 (0.508431911469) cpu_time: 0 (0.1065845) memory: 0 (6799360) fds: 0 (4) utilization: 1 (10.45)
all_kexts avg: duration: 0 (0.506474494934) cpu_time: 0 (0.103775384) memory: 0 (6751232.0) fds: 0 (4.0) utilization: 1 (10.225)
Profiling query: select distinct process.name, listening.port, listening.protocol, listening.family, listening.address, process.pid, process.path, process.on_disk, process.parent, process.start_time from processes as process join listening_ports as listening on process.pid = listening.pid;
processes_binding_to_ports (1/4): duration: 2 (1.50858283043) cpu_time: 2 (1.317375424) memory: 0 (6266880) fds: 0 (4) utilization: 3 (65.7)
processes_binding_to_ports (2/4): duration: 3 (3.01480984688) cpu_time: 2 (2.466791936) memory: 0 (6230016) fds: 0 (4) utilization: 3 (70.2857142857)
processes_binding_to_ports (3/4): duration: 2 (2.51071691513) cpu_time: 2 (2.273139072) memory: 0 (6287360) fds: 0 (4) utilization: 3 (75.5833333333)
processes_binding_to_ports (4/4): duration: 3 (3.01051306725) cpu_time: 2 (2.90141696) memory: 0 (6295552) fds: 0 (4) utilization: 3 (82.7428571429)
processes_binding_to_ports avg: duration: 2 (2.51115566492) cpu_time: 2 (2.239680848) memory: 0 (6269952.0) fds: 0 (4.0) utilization: 3 (73.5779761905)
The results suggest running installed_applications and processes_binding_to_ports less often.
To estimate how often these should run you should evaluate what a differential in the information means from your visibility requirement's perspective (how meaningful is a change vs. how often you check for the change). Then weigh that value against the performance impact of running the query.