New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker support in OSQuery. #3241

Merged
merged 1 commit into from May 5, 2017

Conversation

Projects
None yet
5 participants
@spasam
Contributor

spasam commented May 1, 2017

UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:

  • docker_containers
  • docker_networks
  • docker_volumes
  • docker_images
  • docker_info
  • docker_version

Examples

@theopolis

This comment has been minimized.

Show comment
Hide comment
@theopolis

theopolis May 2, 2017

Contributor

O.o, wow!

Contributor

theopolis commented May 2, 2017

O.o, wow!

Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
}
for (const auto& entry : tree) {
try {

This comment has been minimized.

@theopolis

theopolis May 2, 2017

Contributor

Do we need to wrap this in a try?

@theopolis

theopolis May 2, 2017

Contributor

Do we need to wrap this in a try?

This comment has been minimized.

@spasam

spasam May 2, 2017

Contributor

Nope. Removed

@spasam

spasam May 2, 2017

Contributor

Nope. Removed

Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
@facebook-github-bot

This comment has been minimized.

Show comment
Hide comment
@facebook-github-bot

facebook-github-bot commented May 2, 2017

@spasam updated the pull request - view changes

@spasam

I also added sample output from the tables to the pull request summary. If you have time, take a peek. Thanks

https://github.com/facebook/osquery/files/971156/examples.txt

Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
}
for (const auto& entry : tree) {
try {

This comment has been minimized.

@spasam

spasam May 2, 2017

Contributor

Nope. Removed

@spasam

spasam May 2, 2017

Contributor

Nope. Removed

Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
Show outdated Hide outdated osquery/tables/applications/posix/docker.cpp
@facebook-github-bot

This comment has been minimized.

Show comment
Hide comment
@facebook-github-bot

facebook-github-bot commented May 3, 2017

@spasam updated the pull request - view changes

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 3, 2017

Contributor

I took the liberty of fixing few "make docs" errors in unrelated files.

Contributor

spasam commented May 3, 2017

I took the liberty of fixing few "make docs" errors in unrelated files.

@theopolis

This comment has been minimized.

Show comment
Hide comment
@theopolis

theopolis May 3, 2017

Contributor

Looking good! Will deep dive later today and pull/test locally, one observation before I can provide a complete review, is it possible to mimic the processes table in docker_processes?

Contributor

theopolis commented May 3, 2017

Looking good! Will deep dive later today and pull/test locally, one observation before I can provide a complete review, is it possible to mimic the processes table in docker_processes?

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 3, 2017

Contributor

@theopolis Thanks. Docker processes output is based on the query string arguments. Docker daemon runs "ps" with provided arguments and returns the results. I tried to be as close to "processes" as possible. I might be missing few things. Let me look again.

osquery> select * from processes limit 1;
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| pid | name    | path | cmdline    | state | cwd | root | uid | gid | euid | egid | suid | sgid | on_disk | wired_size | resident_size | total_size | user_time | system_time | start_time | parent | pgroup | threads | nice |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| 1   | systemd |      | /sbin/init | S     |     |      | 0   | 0   | 0    | 0    | 0    | 0    | -1      | 0          | 6108000       | 185456000  | 118       | 160         | 0          | 0      | 1      | 1       | 0    |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+

osquery> select * from docker_container_processes where id = '73';
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| id | pid   | ppid  | pgid  | state | user | uid | gid | euid | egid | suid | sgid | elapsed | nice | threads | rss  | vsize | time     | cpu | mem | command | args            |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| 73 | 14760 | 14742 | 14760 | S     | root | 0   | 0   | 0    | 0    | 0    | 0    | 41      | 0    | 1       | 3324 | 18452 | 00:00:00 | 0.0 | 0.0 | bash    | /bin/bash -x -i |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+

Contributor

spasam commented May 3, 2017

@theopolis Thanks. Docker processes output is based on the query string arguments. Docker daemon runs "ps" with provided arguments and returns the results. I tried to be as close to "processes" as possible. I might be missing few things. Let me look again.

osquery> select * from processes limit 1;
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| pid | name    | path | cmdline    | state | cwd | root | uid | gid | euid | egid | suid | sgid | on_disk | wired_size | resident_size | total_size | user_time | system_time | start_time | parent | pgroup | threads | nice |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+
| 1   | systemd |      | /sbin/init | S     |     |      | 0   | 0   | 0    | 0    | 0    | 0    | -1      | 0          | 6108000       | 185456000  | 118       | 160         | 0          | 0      | 1      | 1       | 0    |
+-----+---------+------+------------+-------+-----+------+-----+-----+------+------+------+------+---------+------------+---------------+------------+-----------+-------------+------------+--------+--------+---------+------+

osquery> select * from docker_container_processes where id = '73';
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| id | pid   | ppid  | pgid  | state | user | uid | gid | euid | egid | suid | sgid | elapsed | nice | threads | rss  | vsize | time     | cpu | mem | command | args            |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+
| 73 | 14760 | 14742 | 14760 | S     | root | 0   | 0   | 0    | 0    | 0    | 0    | 41      | 0    | 1       | 3324 | 18452 | 00:00:00 | 0.0 | 0.0 | bash    | /bin/bash -x -i |
+----+-------+-------+-------+-------+------+-----+-----+------+------+------+------+---------+------+---------+------+-------+----------+-----+-----+---------+-----------------+

@facebook-github-bot

This comment has been minimized.

Show comment
Hide comment
@facebook-github-bot

facebook-github-bot commented May 3, 2017

@spasam updated the pull request - view changes

@theopolis

This comment has been minimized.

Show comment
Hide comment
@theopolis

theopolis May 4, 2017

Contributor

ok to test

Contributor

theopolis commented May 4, 2017

ok to test

@osqueryer

This comment has been minimized.

Show comment
Hide comment
@osqueryer

osqueryer May 4, 2017

Collaborator

👎 The commit d8d687c (Job results: 778) failed one or more tests (macOS/OS X).

Collaborator

osqueryer commented May 4, 2017

👎 The commit d8d687c (Job results: 778) failed one or more tests (macOS/OS X).

@osqueryer

This comment has been minimized.

Show comment
Hide comment
@osqueryer

osqueryer May 4, 2017

Collaborator

👎 The commit d8d687c (Job results: 4614) failed one or more tests (Linux).

Collaborator

osqueryer commented May 4, 2017

👎 The commit d8d687c (Job results: 4614) failed one or more tests (Linux).

@osqueryer

This comment has been minimized.

Show comment
Hide comment
@osqueryer

osqueryer May 4, 2017

Collaborator

👎 The commit d8d687c (Job results: 4615) failed one or more tests (Linux).

Collaborator

osqueryer commented May 4, 2017

👎 The commit d8d687c (Job results: 4615) failed one or more tests (Linux).

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 4, 2017

Contributor

@theopolis Thanks for triggering tests. Anything I have to do to resolve these failures. Looks like workspace is not clean!

error: Your local changes to the following files would be overwritten by merge:
	Library/Homebrew/keg.rb
Please, commit your changes or stash them before you can merge.
Aborting
Contributor

spasam commented May 4, 2017

@theopolis Thanks for triggering tests. Anything I have to do to resolve these failures. Looks like workspace is not clean!

error: Your local changes to the following files would be overwritten by merge:
	Library/Homebrew/keg.rb
Please, commit your changes or stash them before you can merge.
Aborting
@theopolis

This comment has been minimized.

Show comment
Hide comment
@theopolis

theopolis May 4, 2017

Contributor

Yeah, looks like our previous break-fix was actually a timebomb-bug in disguise. This should fix it: #3244 will need to rebase when that lands.

Contributor

theopolis commented May 4, 2017

Yeah, looks like our previous break-fix was actually a timebomb-bug in disguise. This should fix it: #3244 will need to rebase when that lands.

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 4, 2017

Contributor

That was fast. Just saw the other pull request. Thanks!

Contributor

spasam commented May 4, 2017

That was fast. Just saw the other pull request. Thanks!

@facebook-github-bot

This comment has been minimized.

Show comment
Hide comment
@facebook-github-bot

facebook-github-bot commented May 4, 2017

@spasam updated the pull request - view changes

@theopolis

This comment has been minimized.

Show comment
Hide comment
@theopolis

theopolis May 4, 2017

Contributor

Looks like this is now pulling in some unrelated commits.

Can you squash all of your commits and remove the previously-landed commits from this branch?

Contributor

theopolis commented May 4, 2017

Looks like this is now pulling in some unrelated commits.

Can you squash all of your commits and remove the previously-landed commits from this branch?

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 4, 2017

Contributor

Yeah, sorry. I did a merge from upstream and pushed. Not sure what the work flow was to trigger the builds again with your build fix. Will squash!.

Contributor

spasam commented May 4, 2017

Yeah, sorry. I did a merge from upstream and pushed. Not sure what the work flow was to trigger the builds again with your build fix. Will squash!.

Seshu Pasam
Docker support in OSQuery.
UNIX domain socket provided by docker daemon is used to make API calls.
Docker Engine API (v1.27) is used as reference. No support is added for
docker swarm related APIs. This should work on all platforms where
docker UNIX domain socket is exposed.

Supports following top level tables:
- docker_info
- docker_version
- docker_containers
- docker_container_labels
- docker_container_mounts
- docker_container_networks
- docker_container_ports
- docker_images
- docker_image_labels
- docker_networks
- docker_network_labels
- docker_volumes
- docker_volume_labels

Following tables require WHERE clause with container ID:
- docker_container_processes
- docker_container_stats

docker_container_processes almost resembles process table with the
following exceptions:

docker_container_processes does not have following columns
- path
- cwd
- root
- on_disk
- user_time
- system_time

docker_container_processes has these additional columns:
- id (docker container id)
- user
- time
- cpu
- mem

Unrelated to docker changes, I fixed few documentation errors for
"make docs" to pass.
@facebook-github-bot

This comment has been minimized.

Show comment
Hide comment
@facebook-github-bot

facebook-github-bot commented May 4, 2017

@spasam updated the pull request - view changes

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 4, 2017

Contributor

Reverted unrelated commits and squashed docker commits. Updated commit message.

Contributor

spasam commented May 4, 2017

Reverted unrelated commits and squashed docker commits. Updated commit message.

@muffins

muffins approved these changes May 4, 2017

LGTM 💃

@spasam

This comment has been minimized.

Show comment
Hide comment
@spasam

spasam May 5, 2017

Contributor

@theopolis Need anything else for me?

Contributor

spasam commented May 5, 2017

@theopolis Need anything else for me?

@theopolis theopolis merged commit 4cfb314 into facebook:master May 5, 2017

4 checks passed

Code Audit Build finished.
Details
Linux Build finished.
Details
Windows Build finished.
Details
macOS/OS X Build finished.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment