Table of Contents

• Introduction
  • Version
• How it works
• Use it
  • Deploy
    • On docker instance
      • If you do not have a License
      • If you have a Splunk Enterprise License
    • On docker swarm
    • On kubernetes
  • Examples after setup
    • Install application on SHC using SHC Deployer
• TODO

Introduction

NOTE: I'm working at Splunk, but this is not an official Splunk images. I build them in my free time when I'm not at work. I have some knowledge about Splunk, but you should think twice before putting them in production. I run these images on my own home server just for my personal needs. If you have any issues - feel free to open a bug.

Use for learning purposes.

This repository contains set of examples how to run Splunk Enterprise cluster in Docker, including Search Head Cluster and Indexing Cluster.

The main purpose of this repository is to show how to automate Splunk Cluster deployment. Below you can find examples how to setup Cluster on Docker, Swarm Mode, Kubernetes (TODO).

Version

Based on

• Version: 6.4.3
• Build: b03109c2bad4

How it works

These examples depend on the custom image, which you can build using ./splunk-cluster/ folder. This image differs from outcoldman/splunk with just one change. It has special splunk_setup.py script, which allows to pre-configure Splunk. This script supports several commands:

• --wait-splunk schema://hostname:mgmt_port [re_server_role1] [re_server_role2] ... [re_server_roleN]. This command will wait till specified url will reply and that /services/server/info will have all roles defined in server_role list.
• --configure - using environment variables append specific configurations. Environment variables can be defined in format
  • CONF[__{app_location}]__{conf}__{stanza}__{key}={value} - this will append key with value under stanza in conf file in local folder under app_location. If app_location is not specified will be written in local folder under $SPLUNK_HOME/etc/system/.
  • CONF[__{app_location}]__meta__{stanza}__{key}={value} - metadata information will be written in local.meta file in metadata folder under app_location. If app_location is not specified will be written in metadata folder under $SPLUNK_HOME/etc/system/.
• --add-licenses {folder} - all licenses will be added from specified location. If no *.lic files can be find in this folder - script will wait for them.
• --shc-autobootstrap {number_of_expected_shc_members} {mgmt_uri} {local_user} {local_password} {service_discovery_url} {service_discovery_user} {service_discovery_password}. Automatically bootstrap SHC when this will be {number_of_expected_shc_members} number of SHC members. Script will use KVStore endpoint specified under {service_discovery_url} to discover other members. {mgmt_uri} will be used to bootstrap members.
• --healthcheck - using environment variables SPLUNK_HEALTHCHECK_{health_check_name}=schema://hostname:port you can define how container will check that current container is healthy. For example to check that splunkd web server is responsive you can use SPLUNK_HEALTHCHECK_SPLUNKD=https://127.0.0.1:8089.

Use it

Pre-requirement

You need to have a base Splunk image, you can use outcoldman/splunk as base image. Build it and tag it as splunk:latest on the machine where you will build this image.

Deploy

On docker instance

NOTE2: If you are using Docker for Mac - it allocates just 2Gb by default, not enough for this demo. Set more. Maybe 8Gb.

cd ./examples/docker

This folder has two docker-compose files. One which does not require License Master and Splunk Enterprise License docker-compose.yml and second is an extension for the first one, which adds License Master node. Makefile in this folder deals with how docker-compose needs to be invoked.

If you have Splunk Enterprise License copy it in this folder (make sure that license files have extension *.lic) and use all commands with -lm suffix.

Build image.

make build[-lm]

Deploy instances.

make deploy[-lm]

Watch for status of deployment:

• Watch for docker-compose logs -f shc-member for the line Successfully bootstrapped this node as the captain with the given servers.. This will mean that SHC is bootstrapped.
• Open Cluster Master web on http://<docker>:9000 and check Indexer Clustering: Master Node page that Indexes are replicated and ready for search.
• Open SHC member and check that you see logs from all instances index="_internal" | stats count by host.

To clean use

make clean[-lm]

On docker swarm

NOTE1: Splunk Enterprise License is required NOTE2: You have to use docker registry to be sure that each instance will have access to images built by you. Or you can publish image on every swarm instance manually. Specify path to your registry with environment variable SPLUNK_CLUSTER_DOCKER_IMAGE_PATH=registry.yourcompany.com/$USER

cd ./examples/docker-swarm-mode

Copy Splunk Enterprise license (if you have) in this folder (make sure that license files have extension *.lic).

Prepare swarm. This command will create 5 docker-machine instances in VirtualBox. 3 of them will be used in Docker Swarm right away, 2 can be added later

make setup

To use Swarm you need to have access to the Docker registry, specify path to registry and path to image using. If you will not specify anything it will publish image to hub.docker.com/u/$USER/splunk-cluster (create your repo at hub.docker.com)

export SPLUNK_CLUSTER_DOCKER_IMAGE_PATH=registry.yourcompany.com/$USER

Login to your registry (if it is required)

docker login registry.yourcompany.com

Build image.

make build

Publish image to your registry

make push

Deploy cluster.

make deploy

You can add two more nodes to the Swarm cluster by invoking

make setup-add-2

To clean splunk cluster (including volumes) use

make clean-all

To clean images (in case if you want to rebuild)

make clean-images

To download image on each docker instance

make download-image

To remove all docker machines use

make setup-clean

To distribute applications with Cluster Master to the Indexers

docker cp my_app $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/splunk/etc/master-apps/
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/master-apps/my_app
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh splunk apply cluster-bundle --auth admin:changeme --answer-yes

To distribute application with SHC Deployer to SHC Members

docker cp my_app $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master):/opt/splunk/etc/shcluster/apps/
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/my_app
docker exec $(docker ps -qa --filter=label=com.docker.swarm.service.name=cluster-master) entrypoint.sh splunk apply shcluster-bundle -restart true --answer-yes -target https://shc-member-01:8089 -auth admin:changeme

docker service create \
    --name cadvisor \
    --mode global \
    --container-label splunk.cluster=cadvisor \
    --label splunk.cluster=cadvisor \
    --network splunk \
    --with-registry-auth \
    --publish 8080:8080 \
    --mount "type=bind,source=/,target=/rootfs,readonly=true" \
    --mount "type=bind,source=/var/run,target=/var/run,readonly=false" \
    --mount "type=bind,source=/sys,target=/sys,readonly=true" \
    --mount "type=bind,source=/var/lib/docker/,target=/var/lib/docker/,readonly=true" \
    $SPLUNK_CLUSTER_DOCKER_IMAGE_PATH/cadvisor \
        -storage_driver=splunk \
        -storage_driver_splunk_insecureskipverify=true \
        -storage_driver_splunk_source=cadvisor \
        -storage_driver_splunk_token=EF211A51-D6AC-4045-8CD6-F730939AC518 \
        -storage_driver_splunk_url=https://cluster-slave:8088

On kubernetes

TODO

Examples after setup

Install application on SHC using SHC Deployer

docker cp ~/Downloads/splunk_app_aws shc-deployer:/opt/splunk/etc/shcluster/apps/
docker exec shc-deployer entrypoint.sh chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/
docker exec shc-deployer entrypoint.sh splunk apply shcluster-bundle -restart true --answer-yes -target https://$(docker ps --filter=label=splunk.cluster=shc-member -q | head -1):8089 -auth admin:changeme