Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker support in OSQuery. #3241

Merged
merged 1 commit into from
May 5, 2017
Merged

Docker support in OSQuery. #3241

merged 1 commit into from
May 5, 2017

Conversation

spasam
Copy link
Contributor

@spasam spasam commented May 1, 2017

UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:

  • docker_containers
  • docker_networks
  • docker_volumes
  • docker_images
  • docker_info
  • docker_version

Examples

@facebook-github-bot facebook-github-bot added the cla signed Automated label: Pull Request author has signed the osquery CLA label May 1, 2017
@theopolis
Copy link
Member

O.o, wow!

#include <boost/algorithm/string/split.hpp>
#include <boost/asio.hpp>
#include <boost/foreach.hpp>
#include <cstdlib>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, Is it possible to move the C++ includes above the boost? And include a \n\n between the two categories.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@theopolis Thanks for the quick review. Will update with changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

#include "osquery/core/json.h"

namespace pt = boost::property_tree;
using boost::asio::local::stream_protocol;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind aliasing this instead of bringing it into local scope?

using stream_protocol = boost::asio::local::stream_protocol;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to use "local" alias

* docker domain is configured to use a different path specify that path.
*/
FLAG(string,
docker_sock_path,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid! How about docker_socket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed

Row r;
r["id"] = tree.get<std::string>("ID", "");
r["containers"] = INTEGER(tree.get<int>("Containers", 0));
r["container_running"] = INTEGER(tree.get<int>("ContainersRunning", 0));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these (running, paused, stopped) be containers_running, etc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed all three

}

for (const auto& entry : tree) {
try {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to wrap this in a try?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. Removed


for (const auto& entry : tree) {
try {
const pt::ptree node = entry.second;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you grab const&?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

const pt::ptree node = entry.second;
Row r;
r["id"] = getValue(node, ids, "Id");
for (const auto& name : node.get_child("Names")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you .count("Names") you can check for existence.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am assuming you want me to check for count("Names") > 0 before iterating over the names?

*/
QueryData genContainerLabels(QueryContext& context) {
return getLabels(context,
std::string("container"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can provide "const strings" here, drop the std::string() for readability.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if (!key_str.empty()) {
key_str.append("%2C"); // comma
}
key_str.append("%22").append(item).append("%22%3Atrue"); // "item":true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems scary, can I inject arbitrary content into the API call with a where clause?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added utility method to validate sha256 hash/prefix.

@facebook-github-bot
Copy link

@spasam updated the pull request - view changes

Copy link
Contributor Author

@spasam spasam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also added sample output from the tables to the pull request summary. If you have time, take a peek. Thanks

https://github.com/facebook/osquery/files/971156/examples.txt

#include <boost/algorithm/string/split.hpp>
#include <boost/asio.hpp>
#include <boost/foreach.hpp>
#include <cstdlib>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

#include "osquery/core/json.h"

namespace pt = boost::property_tree;
using boost::asio::local::stream_protocol;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to use "local" alias

* docker domain is configured to use a different path specify that path.
*/
FLAG(string,
docker_sock_path,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed

Row r;
r["id"] = tree.get<std::string>("ID", "");
r["containers"] = INTEGER(tree.get<int>("Containers", 0));
r["container_running"] = INTEGER(tree.get<int>("ContainersRunning", 0));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed all three

if (!key_str.empty()) {
key_str.append("%2C"); // comma
}
key_str.append("%22").append(item).append("%22%3Atrue"); // "item":true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added utility method to validate sha256 hash/prefix.

}

for (const auto& entry : tree) {
try {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. Removed


for (const auto& entry : tree) {
try {
const pt::ptree node = entry.second;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

const pt::ptree node = entry.second;
Row r;
r["id"] = getValue(node, ids, "Id");
for (const auto& name : node.get_child("Names")) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am assuming you want me to check for count("Names") > 0 before iterating over the names?

*/
QueryData genContainerLabels(QueryContext& context) {
return getLabels(context,
std::string("container"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@facebook-github-bot
Copy link

@spasam updated the pull request - view changes

@spasam
Copy link
Contributor Author

spasam commented May 3, 2017

I took the liberty of fixing few "make docs" errors in unrelated files.

@theopolis
Copy link
Member

Looking good! Will deep dive later today and pull/test locally, one observation before I can provide a complete review, is it possible to mimic the processes table in docker_processes?

@spasam
Copy link
Contributor Author

spasam commented May 3, 2017

@theopolis Thanks. Docker processes output is based on the query string arguments. Docker daemon runs "ps" with provided arguments and returns the results. I tried to be as close to "processes" as possible. I might be missing few things. Let me look again.

osquery> select * from processes limit 1;
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| pid | name    | path | cmdline    | state | cwd | root | uid | gid | euid | egid | suid | sgid | on_disk | wired_size | resident_size | total_size | user_time | system_time | start_time | parent | pgroup | threads | nice |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| 1   | systemd |      | /sbin/init | S     |     |      | 0   | 0   | 0    | 0    | 0    | 0    | -1      | 0          | 6108000       | 185456000  | 118       | 160         | 0          | 0      | 1      | 1       | 0    |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+

osquery> select * from docker_container_processes where id = '73';
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| id | pid   | ppid  | pgid  | state | user | uid | gid | euid | egid | suid | sgid | elapsed | nice | threads | rss  | vsize | time     | cpu | mem | command | args            |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| 73 | 14760 | 14742 | 14760 | S     | root | 0   | 0   | 0    | 0    | 0    | 0    | 41      | 0    | 1       | 3324 | 18452 | 00:00:00 | 0.0 | 0.0 | bash    | /bin/bash -x -i |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+

@facebook-github-bot
Copy link

@spasam updated the pull request - view changes

@theopolis
Copy link
Member

ok to test

@osqueryer
Copy link

👎 The commit d8d687c (Job results: 778) failed one or more tests (macOS/OS X).

@osqueryer
Copy link

👎 The commit d8d687c (Job results: 4614) failed one or more tests (Linux).

@osqueryer
Copy link

👎 The commit d8d687c (Job results: 4615) failed one or more tests (Linux).

@spasam
Copy link
Contributor Author

spasam commented May 4, 2017

@theopolis Thanks for triggering tests. Anything I have to do to resolve these failures. Looks like workspace is not clean!

error: Your local changes to the following files would be overwritten by merge:
	Library/Homebrew/keg.rb
Please, commit your changes or stash them before you can merge.
Aborting

@theopolis
Copy link
Member

Yeah, looks like our previous break-fix was actually a timebomb-bug in disguise. This should fix it: #3244 will need to rebase when that lands.

@spasam
Copy link
Contributor Author

spasam commented May 4, 2017

That was fast. Just saw the other pull request. Thanks!

@facebook-github-bot
Copy link

@spasam updated the pull request - view changes

@theopolis
Copy link
Member

Looks like this is now pulling in some unrelated commits.

Can you squash all of your commits and remove the previously-landed commits from this branch?

@spasam
Copy link
Contributor Author

spasam commented May 4, 2017

Yeah, sorry. I did a merge from upstream and pushed. Not sure what the work flow was to trigger the builds again with your build fix. Will squash!.

UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:
- docker_info
- docker_version
- docker_containers
- docker_container_labels
- docker_container_mounts
- docker_container_networks
- docker_container_ports
- docker_images
- docker_image_labels
- docker_networks
- docker_network_labels
- docker_volumes
- docker_volume_labels

Following tables require WHERE clause with container ID:
- docker_container_processes
- docker_container_stats

docker_container_processes almost resembles process table with the
following exceptions:

docker_container_processes does not have following columns
- path
- cwd
- root
- on_disk
- user_time
- system_time

docker_container_processes has these additional columns:
- id (docker container id)
- user
- time
- cpu
- mem

Unrelated to docker changes, I fixed few documentation errors for
"make docs" to pass.
@facebook-github-bot
Copy link

@spasam updated the pull request - view changes

@spasam
Copy link
Contributor Author

spasam commented May 4, 2017

Reverted unrelated commits and squashed docker commits. Updated commit message.

Copy link
Contributor

@muffins muffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 💃

@spasam
Copy link
Contributor Author

spasam commented May 5, 2017

@theopolis Need anything else for me?

@theopolis theopolis merged commit 4cfb314 into osquery:master May 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla signed Automated label: Pull Request author has signed the osquery CLA needs response virtual tables
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants