Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker support in OSQuery. #3241

Merged
merged 1 commit into from May 5, 2017
Merged

Docker support in OSQuery. #3241

merged 1 commit into from May 5, 2017

Conversation

@spasam
Copy link
Contributor

@spasam spasam commented May 1, 2017

UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:

  • docker_containers
  • docker_networks
  • docker_volumes
  • docker_images
  • docker_info
  • docker_version

Examples

@theopolis
Copy link
Member

@theopolis theopolis commented May 2, 2017

O.o, wow!

#include <boost/algorithm/string/split.hpp>
#include <boost/asio.hpp>
#include <boost/foreach.hpp>
#include <cstdlib>

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

nit, Is it possible to move the C++ includes above the boost? And include a \n\n between the two categories.

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

@theopolis Thanks for the quick review. Will update with changes.

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Done

#include "osquery/core/json.h"

namespace pt = boost::property_tree;
using boost::asio::local::stream_protocol;

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

Do you mind aliasing this instead of bringing it into local scope?

using stream_protocol = boost::asio::local::stream_protocol;

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Changed to use "local" alias

* docker domain is configured to use a different path specify that path.
*/
FLAG(string,
docker_sock_path,

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

Solid! How about docker_socket?

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Changed

Row r;
r["id"] = tree.get<std::string>("ID", "");
r["containers"] = INTEGER(tree.get<int>("Containers", 0));
r["container_running"] = INTEGER(tree.get<int>("ContainersRunning", 0));

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

Should these (running, paused, stopped) be containers_running, etc?

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Good catch. Fixed all three

}

for (const auto& entry : tree) {
try {

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

Do we need to wrap this in a try?

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Nope. Removed


for (const auto& entry : tree) {
try {
const pt::ptree node = entry.second;

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

Can you grab const&?

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Done

const pt::ptree node = entry.second;
Row r;
r["id"] = getValue(node, ids, "Id");
for (const auto& name : node.get_child("Names")) {

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

If you .count("Names") you can check for existence.

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Am assuming you want me to check for count("Names") > 0 before iterating over the names?

*/
QueryData genContainerLabels(QueryContext& context) {
return getLabels(context,
std::string("container"),

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

You can provide "const strings" here, drop the std::string() for readability.

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Done

if (!key_str.empty()) {
key_str.append("%2C"); // comma
}
key_str.append("%22").append(item).append("%22%3Atrue"); // "item":true

This comment has been minimized.

@theopolis

theopolis May 2, 2017
Member

This seems scary, can I inject arbitrary content into the API call with a where clause?

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

I added utility method to validate sha256 hash/prefix.

@facebook-github-bot
Copy link

@facebook-github-bot facebook-github-bot commented May 2, 2017

@spasam updated the pull request - view changes

Copy link
Contributor Author

@spasam spasam left a comment

I also added sample output from the tables to the pull request summary. If you have time, take a peek. Thanks

https://github.com/facebook/osquery/files/971156/examples.txt

#include <boost/algorithm/string/split.hpp>
#include <boost/asio.hpp>
#include <boost/foreach.hpp>
#include <cstdlib>

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Done

#include "osquery/core/json.h"

namespace pt = boost::property_tree;
using boost::asio::local::stream_protocol;

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Changed to use "local" alias

* docker domain is configured to use a different path specify that path.
*/
FLAG(string,
docker_sock_path,

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Changed

Row r;
r["id"] = tree.get<std::string>("ID", "");
r["containers"] = INTEGER(tree.get<int>("Containers", 0));
r["container_running"] = INTEGER(tree.get<int>("ContainersRunning", 0));

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Good catch. Fixed all three

if (!key_str.empty()) {
key_str.append("%2C"); // comma
}
key_str.append("%22").append(item).append("%22%3Atrue"); // "item":true

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

I added utility method to validate sha256 hash/prefix.

}

for (const auto& entry : tree) {
try {

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Nope. Removed


for (const auto& entry : tree) {
try {
const pt::ptree node = entry.second;

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Done

const pt::ptree node = entry.second;
Row r;
r["id"] = getValue(node, ids, "Id");
for (const auto& name : node.get_child("Names")) {

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Am assuming you want me to check for count("Names") > 0 before iterating over the names?

*/
QueryData genContainerLabels(QueryContext& context) {
return getLabels(context,
std::string("container"),

This comment has been minimized.

@spasam

spasam May 2, 2017
Author Contributor

Done

@facebook-github-bot
Copy link

@facebook-github-bot facebook-github-bot commented May 3, 2017

@spasam updated the pull request - view changes

@spasam
Copy link
Contributor Author

@spasam spasam commented May 3, 2017

I took the liberty of fixing few "make docs" errors in unrelated files.

@theopolis
Copy link
Member

@theopolis theopolis commented May 3, 2017

Looking good! Will deep dive later today and pull/test locally, one observation before I can provide a complete review, is it possible to mimic the processes table in docker_processes?

@spasam
Copy link
Contributor Author

@spasam spasam commented May 3, 2017

@theopolis Thanks. Docker processes output is based on the query string arguments. Docker daemon runs "ps" with provided arguments and returns the results. I tried to be as close to "processes" as possible. I might be missing few things. Let me look again.

osquery> select * from processes limit 1;
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| pid | name    | path | cmdline    | state | cwd | root | uid | gid | euid | egid | suid | sgid | on_disk | wired_size | resident_size | total_size | user_time | system_time | start_time | parent | pgroup | threads | nice |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| 1   | systemd |      | /sbin/init | S     |     |      | 0   | 0   | 0    | 0    | 0    | 0    | -1      | 0          | 6108000       | 185456000  | 118       | 160         | 0          | 0      | 1      | 1       | 0    |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+

osquery> select * from docker_container_processes where id = '73';
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| id | pid   | ppid  | pgid  | state | user | uid | gid | euid | egid | suid | sgid | elapsed | nice | threads | rss  | vsize | time     | cpu | mem | command | args            |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| 73 | 14760 | 14742 | 14760 | S     | root | 0   | 0   | 0    | 0    | 0    | 0    | 41      | 0    | 1       | 3324 | 18452 | 00:00:00 | 0.0 | 0.0 | bash    | /bin/bash -x -i |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+

@facebook-github-bot
Copy link

@facebook-github-bot facebook-github-bot commented May 3, 2017

@spasam updated the pull request - view changes

@theopolis
Copy link
Member

@theopolis theopolis commented May 4, 2017

ok to test

@osqueryer
Copy link

@osqueryer osqueryer commented May 4, 2017

👎 The commit d8d687c (Job results: 778) failed one or more tests (macOS/OS X).

@osqueryer
Copy link

@osqueryer osqueryer commented May 4, 2017

👎 The commit d8d687c (Job results: 4614) failed one or more tests (Linux).

@osqueryer
Copy link

@osqueryer osqueryer commented May 4, 2017

👎 The commit d8d687c (Job results: 4615) failed one or more tests (Linux).

@spasam
Copy link
Contributor Author

@spasam spasam commented May 4, 2017

@theopolis Thanks for triggering tests. Anything I have to do to resolve these failures. Looks like workspace is not clean!

error: Your local changes to the following files would be overwritten by merge:
	Library/Homebrew/keg.rb
Please, commit your changes or stash them before you can merge.
Aborting
@theopolis
Copy link
Member

@theopolis theopolis commented May 4, 2017

Yeah, looks like our previous break-fix was actually a timebomb-bug in disguise. This should fix it: #3244 will need to rebase when that lands.

@spasam
Copy link
Contributor Author

@spasam spasam commented May 4, 2017

That was fast. Just saw the other pull request. Thanks!

@facebook-github-bot
Copy link

@facebook-github-bot facebook-github-bot commented May 4, 2017

@spasam updated the pull request - view changes

@theopolis
Copy link
Member

@theopolis theopolis commented May 4, 2017

Looks like this is now pulling in some unrelated commits.

Can you squash all of your commits and remove the previously-landed commits from this branch?

@spasam
Copy link
Contributor Author

@spasam spasam commented May 4, 2017

Yeah, sorry. I did a merge from upstream and pushed. Not sure what the work flow was to trigger the builds again with your build fix. Will squash!.

UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:
- docker_info
- docker_version
- docker_containers
- docker_container_labels
- docker_container_mounts
- docker_container_networks
- docker_container_ports
- docker_images
- docker_image_labels
- docker_networks
- docker_network_labels
- docker_volumes
- docker_volume_labels

Following tables require WHERE clause with container ID:
- docker_container_processes
- docker_container_stats

docker_container_processes almost resembles process table with the
following exceptions:

docker_container_processes does not have following columns
- path
- cwd
- root
- on_disk
- user_time
- system_time

docker_container_processes has these additional columns:
- id (docker container id)
- user
- time
- cpu
- mem

Unrelated to docker changes, I fixed few documentation errors for
"make docs" to pass.
@spasam spasam force-pushed the spasam:master branch from cdfa3e7 to 1f4f6ca May 4, 2017
@facebook-github-bot
Copy link

@facebook-github-bot facebook-github-bot commented May 4, 2017

@spasam updated the pull request - view changes

@spasam
Copy link
Contributor Author

@spasam spasam commented May 4, 2017

Reverted unrelated commits and squashed docker commits. Updated commit message.

@muffins
muffins approved these changes May 4, 2017
Copy link
Contributor

@muffins muffins left a comment

LGTM 💃

@spasam
Copy link
Contributor Author

@spasam spasam commented May 5, 2017

@theopolis Need anything else for me?

@theopolis theopolis merged commit 4cfb314 into osquery:master May 5, 2017
4 checks passed
4 checks passed
Code Audit Build finished.
Details
Linux Build finished.
Details
Windows Build finished.
Details
macOS/OS X Build finished.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.