Add Splunk logging driver #16207 #16488

outcoldman · Sep 22, 2015

Allow to send Splunk logs using Http Event Collector

Signed-off-by: Denis Gladkikh denis@gladkikh.email

Fixes #16207

glennblock · Sep 22, 2015

This PR is for #16207

tiborvass · Sep 22, 2015

Updated OP's description to include the issue it fixes (so that it automatically closes it if this PR is merged)

albers · Sep 23, 2015

albers reviewed Sep 23, 2015

View changes

albers · Sep 23, 2015

contrib/completion/bash/docker

@@ -270,6 +270,7 @@ __docker_log_drivers() {
 		json-file
 		none
 		syslog
+		splunk


Please insert this in alphabetical order. This also applies to the other additions.

albers · Sep 23, 2015

albers reviewed Sep 23, 2015

View changes

albers · Sep 23, 2015

contrib/completion/bash/docker

@@ -280,10 +281,11 @@ __docker_log_driver_options() {
 	local json_file_options="max-file max-size"
 	local syslog_options="syslog-address syslog-facility tag"
 	local awslogs_options="awslogs-region awslogs-group awslogs-stream"
+	local splunk_options="splunk-url splunk-token splunk-source splunk-sourcetype splunk-index splunk-capath splunk-caname splunk-insecureskipverify"


Please order alphabetically.

albers · Sep 23, 2015

albers reviewed Sep 23, 2015

View changes

albers · Sep 23, 2015

contrib/completion/bash/docker

@@ -347,6 +352,16 @@ __docker_complete_log_driver_options() {
 			" -- "${cur#=}" ) )
 			return
 			;;
+		*splunk-url=*)
+			COMPREPLY=( $( compgen -W "http https" -S "://" -- "${cur#=}" ) )


This does not behave as expected because : is special (contained in COMP_WORDBREAKS).
docker run --log-opt splunk-url=http: will not complete anything.

Please change to

COMPREPLY=( $( compgen -W "http:// https://" -- "${cur#=}" ) ) compopt -o nospace __ltrim_colon_completions "${cur}" return ;;

@albers weird, I took gelf-address and syslog-address as examples, see https://github.com/docker/docker/pull/16488/files#diff-754420541b12dbcbbea00825942a4635R326

albers · Sep 23, 2015

Thanks very much for also implementing bash completion. That's really cool stuff!

outcoldman · Sep 23, 2015

@albers thank you for review! Just updated PR

albers · Sep 23, 2015

Bash completion LGTM. Thanks for updating.

tiborvass · Sep 23, 2015

Didn't read the code, but I'm +1 for a splunk driver since we already have many similar logging drivers.
I do have to state again the fact that it's unfortunate we have to compile all of these drivers in, and would prefer to have these as run-time Go plugins whenever they are ready. I understand having out-of-process logging plugins can be a significant performance hit.

albers · Sep 24, 2015

ping @sdurrheimer zsh completion: new log-driver

LK4D4 · Sep 25, 2015

@outcoldman Is there any easy way to test driver?

outcoldman · Sep 25, 2015

@LK4D4 it depends on the definition of "easy". I did all my tests manually. To test it we need to

Download Splunk
Extract it somewhere, in case if it will be inside docker container - we will need to have access to any mapped volume with supported file system.
The test scripts can preconfigure HTTP Event Collector and start Splunk.
After we can just send some logs to Splunk and verify that these events were indexed.

Do you think that we can perform first 2 steps? Do you have an example?

LK4D4 · Sep 25, 2015

@outcoldman I mean test manually, not unit-tests. Yes, I thought maybe you have docker image with splunk.

outcoldman · Sep 25, 2015

Ah, I see, so you are asking for instructions how to test it manually. This make sense.

You can use my own Docker Image with Splunk (https://hub.docker.com/r/outcoldman/splunk/) - this is not officially supported by Splunk image, but will be good enough for testing. Or you can download Splunk directly in Docker Dev image in mapped volume, let's say at /opt/splunk, you can download it from http://www.splunk.com/en_us/download.html
You need to expose port 8000 for Web Interface and 8088 for HTTP Event Collector.
After you will start Splunk open 8000 to get access to the Web Interface, you will need to configure HTTP Event Collector, see http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/UsetheHTTPEventCollector (use port 8088 for HTTP Event Collector, it is a default port, you can use HTTP or HTTPS, if you want to use HTTPS - you will need to configure custom certificates or use ca certificate for Docker daemon to configure Splunk driver or just ignore verification).
I used these parameters for HTTP (let me know if you also want to test HTTPS) to start nginx container (replace path to your Splunk instance and token given to you by Splunk on previous step)

docker run --log-driver=splunk --log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 --log-opt splunk-url=http://splunk:8088  -d -p 80:80 nginx

After to test the driver I have send few curl requests to nginx container using

curl http://localhost:80/?justatest=hello

In Splunk Web Interface in search string just run "*" and you should see your events.

outcoldman · Sep 25, 2015

outcoldman reviewed Sep 25, 2015

View changes

outcoldman · Sep 25, 2015

daemon/logger/splunk/splunk.go

@@ -0,0 +1,256 @@
+// +build linux


@LK4D4 one question about this line. I saw it in other logging drivers. What is the purpose of this line? As we don't use anything special - our driver should work everywhere where you can run the daemon.
Is it some kind of transition to Windows? Our driver should work on Windows out of box. Should I remove it?

Yup, you should remove it then.
Actually let's ask @jhowardmsft too :)

Yeah, if it works on Windows and you've verified it does, then remove the line. But you'll need a line added to daemon\logdrivers_windows.go too to add splunk.

Supporting Windows is not critical right now as the WIndows support for Docker is till being developed. Once it settles, we'll revisit. We certainly would like long term for it to work on Windows.

Ok, so per @glennblock we do not support Windows right now. I mean it should work, but we have not tested it.
Should I keep this line for now?
Btw, do I need to have special file "splunk_unsupported.go"?

@jhowardmsft the driver technically should work, but the test effort for us to verify on Windows is pretty significant today, based on our recent eval. Is this something that you guys could help us with? We'd be happy to get you a Splunk instance you could test pushing to.

@LK4D4 the Splunk instance can be running on any platform. The driver is just making simple HTTP requests, so I don't see why it would not work on Windows.

@glennblock - Would be happy to verify, just not immediately, busy getting the last few pieces in for technical preview 4 of Windows Server 2016. Drop me a line.

Will do, thanks @jhowardmsft!

Let's remove tag now. It'll work on windows as well.

LK4D4 · Sep 25, 2015

@outcoldman thanks, I'll try. Moving to code-review in the meantime.

LK4D4 · Sep 29, 2015

btw need rebase, sorry :)

outcoldman · Sep 30, 2015

@LK4D4 np, rebased!

halr9000 · Sep 30, 2015

halr9000 reviewed Sep 30, 2015

View changes

halr9000 · Sep 30, 2015

docs/reference/api/docker_remote_api_v1.21.md

@@ -304,7 +304,7 @@ Json Parameters:
        systems, such as SELinux.
    -   **LogConfig** - Log configuration for the container, specified as a JSON object in the form
          `{ "Type": "<driver_name>", "Config": {"key1": "val1"}}`.
-          Available types: `json-file`, `syslog`, `journald`, `gelf`, `awslogs`, `none`.
+          Available types: `json-file`, `syslog`, `journald`, `gelf`, `awslogs`, `splunk`, `none`.


This is getting long. Maybe it should be sorted alpha, with "none" remaining at the end?

hm, yes, we can do that; possible none as the first. I'm fine with leaving that for another PR though.

@outcoldman since this won't make it into docker 1.9, these changes should be moved to docker_remote_api_v1.22.md now

@thaJeztah good point, will move it to 1.22

phemmer · Sep 30, 2015

I would recommend accepting both username & password, as well as http event collector tokens.
As it is, I am unable to test this PR because I am using splunk cloud, which does not yet support the http event collector.
Also, anyone using a self-hosted splunk which is older than 6.3 won't be able to use them either.

glennblock · Sep 30, 2015

Hi @phemmer

Thanks for the input. Event Collector is available in Splunk Cloud, but it needs to be enabled via a support request. There's no UI yet for self-service. But it is definitely available. Please contact Splunk Cloud support and they will get you up and running.

As to the username / password. Going forward we will encourage people not to use the REST API to send events as it requires a username and password that can potentially leak access to Splunk. Also EC has been designed in a more scalable / available fashion than the receiver endpoints. Event Collector tokens are not attached to user credentials, they are specifically designed just for logging events. It is a special endpoint that only allows sending data with a valid token. This makes it more secure from a Splunk perspective as using the token will not allow accessing any part of Splunk, you can't even authenticate to the management port (8089) with the token.

Does that make sense?

You are correct that it is only in Splunk 6.3, but you can stand up dedicated Event Collector instances within a 6.2 environment and they will work fine. EC can run anywhere, an indexer, a forwarder, etc.

thaJeztah · Oct 23, 2015

@moxiegirl gave me a "LGTM" in person 👍 waiting for Janky to finish

glennblock · Oct 23, 2015

Sweet!
On Fri, Oct 23, 2015 at 10:30 AM Sebastiaan van Stijn <
notifications@github.com> wrote:

@moxiegirl https://github.com/moxiegirl gave me a "LGTM" in person [image:
👍] waiting for Janky to finish

—
Reply to this email directly or view it on GitHub
#16488 (comment).

moxiegirl · Oct 23, 2015

I'd recommend clarifying that one paragraph. Otherwise, LGTM

thaJeztah · Oct 23, 2015

Oh, boy.. looks like this needs a rebase :'( sorry for that @outcoldman

edit: sorry for the wrong ping @mountkin, autocomplete hit me :(

outcoldman · Oct 23, 2015

@thaJeztah np, rebased!

thaJeztah · Oct 24, 2015

Al green! merging \o/

LK4D4 · Oct 24, 2015

@outcoldman @glennblock congrats!

outcoldman · Oct 24, 2015

Thank you all for helping! And my personal huge thank to the @LK4D4.
Docker folks, you are great!

thaJeztah · Oct 24, 2015

Thanks for contributing @outcoldman open source FTW!

glennblock · Oct 24, 2015

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!

—
Reply to this email directly or view it on GitHub
#16488 (comment).

glennblock · Oct 24, 2015

Thank you all! We are seeing a lot of interest in Docker in our Splunk
customer base. This is an awesome day!
On Sat, Oct 24, 2015 at 9:11 AM Glenn Block glenn.block@gmail.com wrote:

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!

—
Reply to this email directly or view it on GitHub
#16488 (comment).

pandrew · Oct 24, 2015

Keewl! =)

moxiegirl · Dec 10, 2015

@thaJeztah Looks like this was missed and not cherry picked in. Any idea when 1.9.3 is going out?

LK4D4 · Dec 10, 2015

@moxiegirl I dunno why you think it should be in a milestone. It's a new feature == shouldn't be in the minor release.

thaJeztah · Dec 11, 2015

There was some confusion about the timing, but this PR was merged after the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th).

This driver will be part of the 1.10 release. For people wanting to give it a spin before that, it is available in the "Experimental" or "Master" builds (not intended for production)

glennblock · Dec 11, 2015

Our understanding was it was part of 1.10. Thanks for confirming.

On Thu, Dec 10, 2015 at 4:48 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

There was some confusion about the timing, but this PR was merged after
the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th
https://github.com/docker/docker/releases/tag/v1.9.0-rc1).

This driver will be part of the 1.10 release. For people wanting to give
it a spin before that, it is available in the "Experimental" or "Master"
builds (not intended for production)

—
Reply to this email directly or view it on GitHub
#16488 (comment).

juniorjbn · Aug 17, 2016

Maybe the splunk driver could receive more than one host in "--log-opt splunk-url" for the cases splunk running in cluster ?

glennblock · Aug 17, 2016

Thanks for the suggestion. With HEC the recommendation is always to have a load balancer in front of the HEC pool. Based on that I don't think we need clustering support. Also that will complicate the implementation.

GordonTheTurtle added the status/0-triage label Sep 22, 2015

bfirsh added status/1-design-review and removed status/0-triage labels Sep 23, 2015

GordonTheTurtle added the dco/no label Sep 24, 2015

outcoldman force-pushed the splunk:splunk_log_driver branch from 0c6955f to 5c8a50b Sep 24, 2015

GordonTheTurtle removed the dco/no label Sep 24, 2015

LK4D4 added status/2-code-review and removed status/1-design-review labels Sep 25, 2015

outcoldman force-pushed the splunk:splunk_log_driver branch from b6f7df6 to 9f9c075 Sep 30, 2015

Add Splunk logging driver #16207
Allow to send Splunk logs using Http Event Collector Signed-off-by: Denis Gladkikh <denis@gladkikh.email>

Loading status checks…

1f1dbf3

outcoldman force-pushed the splunk:splunk_log_driver branch from ca2d315 to 1f1dbf3 Oct 23, 2015

thaJeztah added a commit that referenced this pull request Oct 24, 2015

Merge pull request #16488 from splunk/splunk_log_driver
Add Splunk logging driver #16207

8d4888d

glennblock referenced this pull request Oct 25, 2015
Closed
Splunk driver #16207

moxiegirl added this to the 1.9.2 milestone Dec 10, 2015

thaJeztah modified the milestones: 1.10, 1.9.2 Dec 11, 2015

bfirsh referenced this pull request Dec 11, 2015
Closed
Logging driver plugins #18604

thaJeztah added the impact/changelog label Jan 15, 2016

djenriquez referenced this pull request Feb 5, 2016
Closed
Support for Splunk logging driver #304

outcoldman deleted the splunk:splunk_log_driver branch Aug 25, 2016

moby/moby

Join GitHub today

Add Splunk logging driver #16207 #16488

Conversation

outcoldman commented Sep 22, 2015

GordonTheTurtle added the status/0-triage label Sep 22, 2015

This comment has been minimized.

glennblock commented Sep 22, 2015

This comment has been minimized.

tiborvass commented Sep 22, 2015

albers reviewed Sep 23, 2015 View changes

This comment has been minimized.

This comment has been minimized.

albers reviewed Sep 23, 2015 View changes

This comment has been minimized.

This comment has been minimized.

albers reviewed Sep 23, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

albers commented Sep 23, 2015

This comment has been minimized.

outcoldman commented Sep 23, 2015

bfirsh added status/1-design-review and removed status/0-triage labels Sep 23, 2015

This comment has been minimized.

albers commented Sep 23, 2015

This comment has been minimized.

tiborvass commented Sep 23, 2015

This comment has been minimized.

albers commented Sep 24, 2015

GordonTheTurtle added the dco/no label Sep 24, 2015

outcoldman force-pushed the splunk:splunk_log_driver branch from 0c6955f to 5c8a50b Sep 24, 2015

GordonTheTurtle removed the dco/no label Sep 24, 2015

This comment has been minimized.

LK4D4 commented Sep 25, 2015

This comment has been minimized.

outcoldman commented Sep 25, 2015

This comment has been minimized.

LK4D4 commented Sep 25, 2015

This comment has been minimized.

outcoldman commented Sep 25, 2015

outcoldman reviewed Sep 25, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

LK4D4 added status/2-code-review and removed status/1-design-review labels Sep 25, 2015

This comment has been minimized.

LK4D4 commented Sep 25, 2015

This comment has been minimized.

LK4D4 commented Sep 29, 2015

outcoldman force-pushed the splunk:splunk_log_driver branch from b6f7df6 to 9f9c075 Sep 30, 2015

This comment has been minimized.

outcoldman commented Sep 30, 2015

halr9000 reviewed Sep 30, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

phemmer commented Sep 30, 2015

This comment has been minimized.

glennblock commented Sep 30, 2015

This comment has been minimized.

thaJeztah commented Oct 23, 2015

This comment has been minimized.

glennblock commented Oct 23, 2015

This comment has been minimized.

moxiegirl commented Oct 23, 2015

This comment has been minimized.

thaJeztah commented Oct 23, 2015

outcoldman force-pushed the splunk:splunk_log_driver branch from ca2d315 to 1f1dbf3 Oct 23, 2015

This comment has been minimized.

outcoldman commented Oct 23, 2015

This comment has been minimized.

albers reviewed Sep 23, 2015

View changes

albers reviewed Sep 23, 2015

View changes

albers reviewed Sep 23, 2015

View changes

outcoldman force-pushed the splunk:splunk_log_driver branch from `0c6955f` to `5c8a50b` Sep 24, 2015

outcoldman reviewed Sep 25, 2015

View changes

outcoldman force-pushed the splunk:splunk_log_driver branch from `b6f7df6` to `9f9c075` Sep 30, 2015

halr9000 reviewed Sep 30, 2015

View changes

outcoldman force-pushed the splunk:splunk_log_driver branch from `ca2d315` to `1f1dbf3` Oct 23, 2015

thaJeztah merged commit `8d4888d` into moby:master Oct 24, 2015
5 checks passed