Docker support in OSQuery. #3241

spasam · 2017-05-01T16:54:33Z

UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:

docker_containers
docker_networks
docker_volumes
docker_images
docker_info
docker_version

Examples

theopolis · 2017-05-02T03:37:48Z

O.o, wow!

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+#include <boost/algorithm/string/split.hpp>
+#include <boost/asio.hpp>
+#include <boost/foreach.hpp>
+#include <cstdlib>


nit, Is it possible to move the C++ includes above the boost? And include a \n\n between the two categories.

@theopolis Thanks for the quick review. Will update with changes.

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+#include "osquery/core/json.h"
+
+namespace pt = boost::property_tree;
+using boost::asio::local::stream_protocol;


Do you mind aliasing this instead of bringing it into local scope?

using stream_protocol = boost::asio::local::stream_protocol;

Changed to use "local" alias

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+ * docker domain is configured to use a different path specify that path.
+ */
+FLAG(string,
+     docker_sock_path,


Solid! How about docker_socket?

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+  Row r;
+  r["id"] = tree.get<std::string>("ID", "");
+  r["containers"] = INTEGER(tree.get<int>("Containers", 0));
+  r["container_running"] = INTEGER(tree.get<int>("ContainersRunning", 0));


Should these (running, paused, stopped) be containers_running, etc?

Good catch. Fixed all three

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+  }
+
+  for (const auto& entry : tree) {
+    try {


Do we need to wrap this in a try?

Nope. Removed

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+
+  for (const auto& entry : tree) {
+    try {
+      const pt::ptree node = entry.second;


Can you grab const&?

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+      const pt::ptree node = entry.second;
+      Row r;
+      r["id"] = getValue(node, ids, "Id");
+      for (const auto& name : node.get_child("Names")) {


If you .count("Names") you can check for existence.

Am assuming you want me to check for count("Names") > 0 before iterating over the names?

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+ */
+QueryData genContainerLabels(QueryContext& context) {
+  return getLabels(context,
+                    std::string("container"),


You can provide "const strings" here, drop the std::string() for readability.

theopolis · 2017-05-02T04:06:34Z

osquery/tables/applications/posix/docker.cpp

+      if (!key_str.empty()) {
+        key_str.append("%2C"); // comma
+      }
+      key_str.append("%22").append(item).append("%22%3Atrue"); // "item":true


This seems scary, can I inject arbitrary content into the API call with a where clause?

I added utility method to validate sha256 hash/prefix.

facebook-github-bot · 2017-05-02T16:47:28Z

@spasam updated the pull request - view changes

spasam

I also added sample output from the tables to the pull request summary. If you have time, take a peek. Thanks

https://github.com/facebook/osquery/files/971156/examples.txt

spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+#include <boost/algorithm/string/split.hpp>
+#include <boost/asio.hpp>
+#include <boost/foreach.hpp>
+#include <cstdlib>


spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+#include "osquery/core/json.h"
+
+namespace pt = boost::property_tree;
+using boost::asio::local::stream_protocol;


Changed to use "local" alias

spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+ * docker domain is configured to use a different path specify that path.
+ */
+FLAG(string,
+     docker_sock_path,


spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+  Row r;
+  r["id"] = tree.get<std::string>("ID", "");
+  r["containers"] = INTEGER(tree.get<int>("Containers", 0));
+  r["container_running"] = INTEGER(tree.get<int>("ContainersRunning", 0));


Good catch. Fixed all three

spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+      if (!key_str.empty()) {
+        key_str.append("%2C"); // comma
+      }
+      key_str.append("%22").append(item).append("%22%3Atrue"); // "item":true


I added utility method to validate sha256 hash/prefix.

spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+  }
+
+  for (const auto& entry : tree) {
+    try {


Nope. Removed

spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+
+  for (const auto& entry : tree) {
+    try {
+      const pt::ptree node = entry.second;


spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+      const pt::ptree node = entry.second;
+      Row r;
+      r["id"] = getValue(node, ids, "Id");
+      for (const auto& name : node.get_child("Names")) {


Am assuming you want me to check for count("Names") > 0 before iterating over the names?

spasam · 2017-05-02T16:54:14Z

osquery/tables/applications/posix/docker.cpp

+ */
+QueryData genContainerLabels(QueryContext& context) {
+  return getLabels(context,
+                    std::string("container"),


facebook-github-bot · 2017-05-03T06:53:03Z

@spasam updated the pull request - view changes

spasam · 2017-05-03T06:54:05Z

I took the liberty of fixing few "make docs" errors in unrelated files.

theopolis · 2017-05-03T15:59:32Z

Looking good! Will deep dive later today and pull/test locally, one observation before I can provide a complete review, is it possible to mimic the processes table in docker_processes?

spasam · 2017-05-03T16:07:35Z

@theopolis Thanks. Docker processes output is based on the query string arguments. Docker daemon runs "ps" with provided arguments and returns the results. I tried to be as close to "processes" as possible. I might be missing few things. Let me look again.

osquery> select * from processes limit 1;
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| pid | name    | path | cmdline    | state | cwd | root | uid | gid | euid | egid | suid | sgid | on_disk | wired_size | resident_size | total_size | user_time | system_time | start_time | parent | pgroup | threads | nice |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| 1   | systemd |      | /sbin/init | S     |     |      | 0   | 0   | 0    | 0    | 0    | 0    | -1      | 0          | 6108000       | 185456000  | 118       | 160         | 0          | 0      | 1      | 1       | 0    |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+

osquery> select * from docker_container_processes where id = '73';
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| id | pid   | ppid  | pgid  | state | user | uid | gid | euid | egid | suid | sgid | elapsed | nice | threads | rss  | vsize | time     | cpu | mem | command | args            |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| 73 | 14760 | 14742 | 14760 | S     | root | 0   | 0   | 0    | 0    | 0    | 0    | 41      | 0    | 1       | 3324 | 18452 | 00:00:00 | 0.0 | 0.0 | bash    | /bin/bash -x -i |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+

facebook-github-bot · 2017-05-03T19:57:39Z

@spasam updated the pull request - view changes

theopolis · 2017-05-04T02:39:55Z

ok to test

osqueryer · 2017-05-04T02:40:22Z

👎 The commit d8d687c (Job results: 778) failed one or more tests (macOS/OS X).

osqueryer · 2017-05-04T02:46:43Z

👎 The commit d8d687c (Job results: 4614) failed one or more tests (Linux).

osqueryer · 2017-05-04T02:52:01Z

👎 The commit d8d687c (Job results: 4615) failed one or more tests (Linux).

spasam · 2017-05-04T02:57:16Z

@theopolis Thanks for triggering tests. Anything I have to do to resolve these failures. Looks like workspace is not clean!

error: Your local changes to the following files would be overwritten by merge:
	Library/Homebrew/keg.rb
Please, commit your changes or stash them before you can merge.
Aborting

theopolis · 2017-05-04T03:08:19Z

Yeah, looks like our previous break-fix was actually a timebomb-bug in disguise. This should fix it: #3244 will need to rebase when that lands.

spasam · 2017-05-04T03:13:28Z

That was fast. Just saw the other pull request. Thanks!

facebook-github-bot · 2017-05-04T04:25:48Z

@spasam updated the pull request - view changes

theopolis · 2017-05-04T05:30:56Z

Looks like this is now pulling in some unrelated commits.

Can you squash all of your commits and remove the previously-landed commits from this branch?

spasam · 2017-05-04T05:33:59Z

Yeah, sorry. I did a merge from upstream and pushed. Not sure what the work flow was to trigger the builds again with your build fix. Will squash!.

UNIX domain socket provided by docker daemon is used to make API calls. Docker Engine API (v1.27) is used as reference. No support is added for docker swarm related APIs. This should work on all platforms where docker UNIX domain socket is exposed. Supports following top level tables: - docker_info - docker_version - docker_containers - docker_container_labels - docker_container_mounts - docker_container_networks - docker_container_ports - docker_images - docker_image_labels - docker_networks - docker_network_labels - docker_volumes - docker_volume_labels Following tables require WHERE clause with container ID: - docker_container_processes - docker_container_stats docker_container_processes almost resembles process table with the following exceptions: docker_container_processes does not have following columns - path - cwd - root - on_disk - user_time - system_time docker_container_processes has these additional columns: - id (docker container id) - user - time - cpu - mem Unrelated to docker changes, I fixed few documentation errors for "make docs" to pass.

facebook-github-bot · 2017-05-04T05:47:58Z

@spasam updated the pull request - view changes

spasam · 2017-05-04T13:45:42Z

Reverted unrelated commits and squashed docker commits. Updated commit message.

muffins

LGTM 💃

spasam · 2017-05-05T05:36:51Z

@theopolis Need anything else for me?

facebook-github-bot added the cla signed label May 1, 2017

theopolis added the virtual tables label May 2, 2017

theopolis requested changes May 2, 2017

View changes

spasam reviewed May 2, 2017

View changes

theopolis approved these changes May 4, 2017

View changes

theopolis added the needs response label May 4, 2017

spasam force-pushed the spasam:master branch from cdfa3e7 to 1f4f6ca May 4, 2017

muffins approved these changes May 4, 2017

View changes

theopolis merged commit 4cfb314 into osquery:master May 5, 2017
4 checks passed

4 checks passed

Code Audit Build finished.
Details

Linux Build finished.
Details

Windows Build finished.
Details

macOS/OS X Build finished.
Details

osquery / osquery

Join GitHub today

Docker support in OSQuery. #3241

Docker support in OSQuery. #3241

Conversation

spasam commented May 1, 2017 • edited

theopolis commented May 2, 2017

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

theopolis May 2, 2017 Member

This comment has been minimized.

spasam May 2, 2017 Author Contributor

facebook-github-bot commented May 2, 2017

spasam left a comment

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

This comment has been minimized.

spasam May 2, 2017 Author Contributor

facebook-github-bot commented May 3, 2017

spasam commented May 3, 2017

theopolis commented May 3, 2017

spasam commented May 3, 2017

facebook-github-bot commented May 3, 2017

theopolis commented May 4, 2017

osqueryer commented May 4, 2017

osqueryer commented May 4, 2017

osqueryer commented May 4, 2017

spasam commented May 4, 2017

theopolis commented May 4, 2017

spasam commented May 4, 2017

facebook-github-bot commented May 4, 2017

theopolis commented May 4, 2017

spasam commented May 4, 2017

spasam commented May 1, 2017 •

edited

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

theopolis May 2, 2017
Member

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor

spasam May 2, 2017
Author Contributor