New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Splunk logging driver #16207 #16488

Merged
merged 1 commit into from Oct 24, 2015

Conversation

Projects
None yet
@outcoldman
Copy link
Contributor

outcoldman commented Sep 22, 2015

Allow to send Splunk logs using Http Event Collector

Signed-off-by: Denis Gladkikh denis@gladkikh.email

Fixes #16207

@glennblock

This comment has been minimized.

Copy link

glennblock commented Sep 22, 2015

This PR is for #16207

@tiborvass

This comment has been minimized.

Copy link
Collaborator

tiborvass commented Sep 22, 2015

Updated OP's description to include the issue it fixes (so that it automatically closes it if this PR is merged)

@@ -270,6 +270,7 @@ __docker_log_drivers() {
json-file
none
syslog
splunk

This comment has been minimized.

@albers

albers Sep 23, 2015

Member

Please insert this in alphabetical order. This also applies to the other additions.

This comment has been minimized.

@outcoldman

outcoldman Sep 23, 2015

Author Contributor

Fixed!

@@ -280,10 +281,11 @@ __docker_log_driver_options() {
local json_file_options="max-file max-size"
local syslog_options="syslog-address syslog-facility tag"
local awslogs_options="awslogs-region awslogs-group awslogs-stream"
local splunk_options="splunk-url splunk-token splunk-source splunk-sourcetype splunk-index splunk-capath splunk-caname splunk-insecureskipverify"

This comment has been minimized.

@albers

albers Sep 23, 2015

Member

Please order alphabetically.

This comment has been minimized.

@outcoldman

outcoldman Sep 23, 2015

Author Contributor

Fixed!

@@ -347,6 +352,16 @@ __docker_complete_log_driver_options() {
" -- "${cur#=}" ) )
return
;;
*splunk-url=*)
COMPREPLY=( $( compgen -W "http https" -S "://" -- "${cur#=}" ) )

This comment has been minimized.

@albers

albers Sep 23, 2015

Member

This does not behave as expected because : is special (contained in COMP_WORDBREAKS).
docker run --log-opt splunk-url=http: will not complete anything.

Please change to

COMPREPLY=( $( compgen -W "http:// https://" -- "${cur#=}" ) )
compopt -o nospace
__ltrim_colon_completions "${cur}"
return
;;

This comment has been minimized.

@outcoldman

outcoldman Sep 23, 2015

Author Contributor

@albers weird, I took gelf-address and syslog-address as examples, see https://github.com/docker/docker/pull/16488/files#diff-754420541b12dbcbbea00825942a4635R326

@albers

This comment has been minimized.

Copy link
Member

albers commented Sep 23, 2015

Thanks very much for also implementing bash completion. That's really cool stuff!

@outcoldman

This comment has been minimized.

Copy link
Contributor Author

outcoldman commented Sep 23, 2015

@albers thank you for review! Just updated PR

@albers

This comment has been minimized.

Copy link
Member

albers commented Sep 23, 2015

Bash completion LGTM. Thanks for updating.

@tiborvass

This comment has been minimized.

Copy link
Collaborator

tiborvass commented Sep 23, 2015

Didn't read the code, but I'm +1 for a splunk driver since we already have many similar logging drivers.
I do have to state again the fact that it's unfortunate we have to compile all of these drivers in, and would prefer to have these as run-time Go plugins whenever they are ready. I understand having out-of-process logging plugins can be a significant performance hit.

@albers

This comment has been minimized.

Copy link
Member

albers commented Sep 24, 2015

ping @sdurrheimer zsh completion: new log-driver

@outcoldman outcoldman force-pushed the splunk:splunk_log_driver branch from 0c6955f to 5c8a50b Sep 24, 2015

@GordonTheTurtle GordonTheTurtle removed the dco/no label Sep 24, 2015

@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Sep 25, 2015

@outcoldman Is there any easy way to test driver?

@outcoldman

This comment has been minimized.

Copy link
Contributor Author

outcoldman commented Sep 25, 2015

@LK4D4 it depends on the definition of "easy". I did all my tests manually. To test it we need to

  1. Download Splunk
  2. Extract it somewhere, in case if it will be inside docker container - we will need to have access to any mapped volume with supported file system.
  3. The test scripts can preconfigure HTTP Event Collector and start Splunk.
  4. After we can just send some logs to Splunk and verify that these events were indexed.

Do you think that we can perform first 2 steps? Do you have an example?

@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Sep 25, 2015

@outcoldman I mean test manually, not unit-tests. Yes, I thought maybe you have docker image with splunk.

@outcoldman

This comment has been minimized.

Copy link
Contributor Author

outcoldman commented Sep 25, 2015

Ah, I see, so you are asking for instructions how to test it manually. This make sense.

  1. You can use my own Docker Image with Splunk (https://hub.docker.com/r/outcoldman/splunk/) - this is not officially supported by Splunk image, but will be good enough for testing. Or you can download Splunk directly in Docker Dev image in mapped volume, let's say at /opt/splunk, you can download it from http://www.splunk.com/en_us/download.html
  2. You need to expose port 8000 for Web Interface and 8088 for HTTP Event Collector.
  3. After you will start Splunk open 8000 to get access to the Web Interface, you will need to configure HTTP Event Collector, see http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/UsetheHTTPEventCollector (use port 8088 for HTTP Event Collector, it is a default port, you can use HTTP or HTTPS, if you want to use HTTPS - you will need to configure custom certificates or use ca certificate for Docker daemon to configure Splunk driver or just ignore verification).
  4. I used these parameters for HTTP (let me know if you also want to test HTTPS) to start nginx container (replace path to your Splunk instance and token given to you by Splunk on previous step)
docker run --log-driver=splunk --log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 --log-opt splunk-url=http://splunk:8088  -d -p 80:80 nginx
  1. After to test the driver I have send few curl requests to nginx container using
curl http://localhost:80/?justatest=hello
  1. In Splunk Web Interface in search string just run "*" and you should see your events.
@@ -0,0 +1,256 @@
// +build linux

This comment has been minimized.

@outcoldman

outcoldman Sep 25, 2015

Author Contributor

@LK4D4 one question about this line. I saw it in other logging drivers. What is the purpose of this line? As we don't use anything special - our driver should work everywhere where you can run the daemon.
Is it some kind of transition to Windows? Our driver should work on Windows out of box. Should I remove it?

This comment has been minimized.

@LK4D4

LK4D4 Sep 25, 2015

Contributor

Yup, you should remove it then.
Actually let's ask @jhowardmsft too :)

This comment has been minimized.

@jhowardmsft

jhowardmsft Sep 25, 2015

Contributor

Yeah, if it works on Windows and you've verified it does, then remove the line. But you'll need a line added to daemon\logdrivers_windows.go too to add splunk.

This comment has been minimized.

@glennblock

glennblock Sep 25, 2015

Supporting Windows is not critical right now as the WIndows support for Docker is till being developed. Once it settles, we'll revisit. We certainly would like long term for it to work on Windows.

This comment has been minimized.

@outcoldman

outcoldman Sep 25, 2015

Author Contributor

Ok, so per @glennblock we do not support Windows right now. I mean it should work, but we have not tested it.
Should I keep this line for now?
Btw, do I need to have special file "splunk_unsupported.go"?

This comment has been minimized.

@glennblock

glennblock Sep 25, 2015

@jhowardmsft the driver technically should work, but the test effort for us to verify on Windows is pretty significant today, based on our recent eval. Is this something that you guys could help us with? We'd be happy to get you a Splunk instance you could test pushing to.

This comment has been minimized.

@glennblock

glennblock Sep 25, 2015

@LK4D4 the Splunk instance can be running on any platform. The driver is just making simple HTTP requests, so I don't see why it would not work on Windows.

This comment has been minimized.

@jhowardmsft

jhowardmsft Sep 25, 2015

Contributor

@glennblock - Would be happy to verify, just not immediately, busy getting the last few pieces in for technical preview 4 of Windows Server 2016. Drop me a line.

This comment has been minimized.

@glennblock

This comment has been minimized.

@LK4D4

LK4D4 Oct 21, 2015

Contributor

Let's remove tag now. It'll work on windows as well.

@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Sep 25, 2015

@outcoldman thanks, I'll try. Moving to code-review in the meantime.

@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Sep 29, 2015

btw need rebase, sorry :)

@outcoldman outcoldman force-pushed the splunk:splunk_log_driver branch from b6f7df6 to 9f9c075 Sep 30, 2015

@outcoldman

This comment has been minimized.

Copy link
Contributor Author

outcoldman commented Sep 30, 2015

@LK4D4 np, rebased!

@@ -304,7 +304,7 @@ Json Parameters:
systems, such as SELinux.
- **LogConfig** - Log configuration for the container, specified as a JSON object in the form
`{ "Type": "<driver_name>", "Config": {"key1": "val1"}}`.
Available types: `json-file`, `syslog`, `journald`, `gelf`, `awslogs`, `none`.
Available types: `json-file`, `syslog`, `journald`, `gelf`, `awslogs`, `splunk`, `none`.

This comment has been minimized.

@halr9000

halr9000 Sep 30, 2015

This is getting long. Maybe it should be sorted alpha, with "none" remaining at the end?

This comment has been minimized.

@thaJeztah

thaJeztah Oct 21, 2015

Member

hm, yes, we can do that; possible none as the first. I'm fine with leaving that for another PR though.

@outcoldman since this won't make it into docker 1.9, these changes should be moved to docker_remote_api_v1.22.md now

This comment has been minimized.

@outcoldman

outcoldman Oct 23, 2015

Author Contributor

@thaJeztah good point, will move it to 1.22

@phemmer

This comment has been minimized.

Copy link
Contributor

phemmer commented Sep 30, 2015

I would recommend accepting both username & password, as well as http event collector tokens.
As it is, I am unable to test this PR because I am using splunk cloud, which does not yet support the http event collector.
Also, anyone using a self-hosted splunk which is older than 6.3 won't be able to use them either.

@glennblock

This comment has been minimized.

Copy link

glennblock commented Sep 30, 2015

Hi @phemmer

Thanks for the input. Event Collector is available in Splunk Cloud, but it needs to be enabled via a support request. There's no UI yet for self-service. But it is definitely available. Please contact Splunk Cloud support and they will get you up and running.

As to the username / password. Going forward we will encourage people not to use the REST API to send events as it requires a username and password that can potentially leak access to Splunk. Also EC has been designed in a more scalable / available fashion than the receiver endpoints. Event Collector tokens are not attached to user credentials, they are specifically designed just for logging events. It is a special endpoint that only allows sending data with a valid token. This makes it more secure from a Splunk perspective as using the token will not allow accessing any part of Splunk, you can't even authenticate to the management port (8089) with the token.

Does that make sense?

You are correct that it is only in Splunk 6.3, but you can stand up dedicated Event Collector instances within a 6.2 environment and they will work fine. EC can run anywhere, an indexer, a forwarder, etc.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 23, 2015

@moxiegirl gave me a "LGTM" in person 👍 waiting for Janky to finish

@glennblock

This comment has been minimized.

Copy link

glennblock commented Oct 23, 2015

Sweet!
On Fri, Oct 23, 2015 at 10:30 AM Sebastiaan van Stijn <
notifications@github.com> wrote:

@moxiegirl https://github.com/moxiegirl gave me a "LGTM" in person [image:
👍] waiting for Janky to finish


Reply to this email directly or view it on GitHub
#16488 (comment).

@moxiegirl

This comment has been minimized.

Copy link
Contributor

moxiegirl commented Oct 23, 2015

I'd recommend clarifying that one paragraph. Otherwise, LGTM

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 23, 2015

Oh, boy.. looks like this needs a rebase :'( sorry for that @outcoldman

edit: sorry for the wrong ping @mountkin, autocomplete hit me :(

Add Splunk logging driver #16207
Allow to send Splunk logs using Http Event Collector

Signed-off-by: Denis Gladkikh <denis@gladkikh.email>

@outcoldman outcoldman force-pushed the splunk:splunk_log_driver branch from ca2d315 to 1f1dbf3 Oct 23, 2015

@outcoldman

This comment has been minimized.

Copy link
Contributor Author

outcoldman commented Oct 23, 2015

@thaJeztah np, rebased!

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 24, 2015

Al green! merging \o/

thaJeztah added a commit that referenced this pull request Oct 24, 2015

@thaJeztah thaJeztah merged commit 8d4888d into moby:master Oct 24, 2015

5 checks passed

docker/dco-signed All commits signed
Details
experimental Jenkins build Docker-PRs-experimental 9689 has succeeded
Details
janky Jenkins build Docker-PRs 18454 has succeeded
Details
userns Jenkins build Docker-PRs-userns 831 has succeeded
Details
windows Jenkins build Windows-PRs 15674 has succeeded
Details
@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Oct 24, 2015

@outcoldman

This comment has been minimized.

Copy link
Contributor Author

outcoldman commented Oct 24, 2015

Thank you all for helping! And my personal huge thank to the @LK4D4.
Docker folks, you are great!

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 24, 2015

Thanks for contributing @outcoldman open source FTW!

@glennblock

This comment has been minimized.

Copy link

glennblock commented Oct 24, 2015

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!


Reply to this email directly or view it on GitHub
#16488 (comment).

@glennblock

This comment has been minimized.

Copy link

glennblock commented Oct 24, 2015

Thank you all! We are seeing a lot of interest in Docker in our Splunk
customer base. This is an awesome day!
On Sat, Oct 24, 2015 at 9:11 AM Glenn Block glenn.block@gmail.com wrote:

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!


Reply to this email directly or view it on GitHub
#16488 (comment).

@pandrew

This comment has been minimized.

Copy link
Contributor

pandrew commented Oct 24, 2015

Keewl! =)

@glennblock glennblock referenced this pull request Oct 25, 2015

Closed

Splunk driver #16207

@moxiegirl moxiegirl added this to the 1.9.2 milestone Dec 10, 2015

@moxiegirl

This comment has been minimized.

Copy link
Contributor

moxiegirl commented Dec 10, 2015

@thaJeztah Looks like this was missed and not cherry picked in. Any idea when 1.9.3 is going out?

@LK4D4

This comment has been minimized.

Copy link
Contributor

LK4D4 commented Dec 10, 2015

@moxiegirl I dunno why you think it should be in a milestone. It's a new feature == shouldn't be in the minor release.

@thaJeztah thaJeztah modified the milestones: 1.10, 1.9.2 Dec 11, 2015

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Dec 11, 2015

There was some confusion about the timing, but this PR was merged after the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th).

This driver will be part of the 1.10 release. For people wanting to give it a spin before that, it is available in the "Experimental" or "Master" builds (not intended for production)

@glennblock

This comment has been minimized.

Copy link

glennblock commented Dec 11, 2015

Our understanding was it was part of 1.10. Thanks for confirming.

On Thu, Dec 10, 2015 at 4:48 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

There was some confusion about the timing, but this PR was merged after
the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th
https://github.com/docker/docker/releases/tag/v1.9.0-rc1).

This driver will be part of the 1.10 release. For people wanting to give
it a spin before that, it is available in the "Experimental" or "Master"
builds (not intended for production)


Reply to this email directly or view it on GitHub
#16488 (comment).

@juniorjbn

This comment has been minimized.

Copy link

juniorjbn commented Aug 17, 2016

Maybe the splunk driver could receive more than one host in "--log-opt splunk-url" for the cases splunk running in cluster ?

@glennblock

This comment has been minimized.

Copy link

glennblock commented Aug 17, 2016

Thanks for the suggestion. With HEC the recommendation is always to have a load balancer in front of the HEC pool. Based on that I don't think we need clustering support. Also that will complicate the implementation.

@outcoldman outcoldman deleted the splunk:splunk_log_driver branch Aug 25, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment