Event Streams connector for Splunk

Introduction

Event Streams connector for Splunk reads the logs / events from an IBM Event Streams topic and writes to a Splunk instance. This repo assumes that you are using the Export to Event Streams capability to stream the logs from IBM Log Analysis or IBM Cloud Activity Tracker.

It uses Kafka Connect, Splunk Connect for Kafka deployed on an IBM Kubernetes cluster.

As illustrated, the Event Streams connector for Splunk has following components:

• IBM Cloud Event Streams (as the log / event source)
• IBM Cloud Kubernetes service (to run Kafka Connect & the Splunk Connect for Kafka)
• Splunk (as the log sink)

This repo automates the deployment of the Event Streams connector for Splunk capability using Terraform & IBM Cloud Schematics.

Prerequisites:

1. You must have an IBM Event Streams (Enterprise plan), as a log source
   • The name of the Event Streams instance is your event_stream_name.
   • Configure the Export to Event Streams capability to publish the logs from IBM Cloud Logging to a Event Streams topic event_stream_topic;
2. You must have a Splunk Enterprise server instance already provisioned; and you know the following details about the log sink
   • splunk_index, splunk_hec_uri & splunk_hec_token to write the logs

Deploy Event Streams connector for Splunk using IBM Cloud Schematics

1. Click on Deploy to IBM Cloud to create a Schematics Workspace for Event Streams connector for Splunk. By default, the following parameters will be pre-filled :

   • Repository URL (cannot be changed)
   • IBM Cloud Account (ibmcloud_account, displayed on the right-top corner of the IBM Cloud Console - can be changed)
   • Workspace name (workspace_name - can be changed)
   • Workspace tags (workspace_tags - can be changed)
   • Workspace resource group (workspace_resource_group for the workspace - can be changed)
   • Workspace location (workspace_region for the workspace - can be changed)

   Note: The workspace_tags, workspace_resource_group & workspace_region - is independent, and is not related to the Logging & Event Streams services parameters.

2. Click the "create".

3. Review the following input variables, and update them according to your needs

   Input variable	Description	Type	Default	Required ?	Sensitive ?
   ibmcloud_api_key	Enter your IBM Cloud API Key, you can get your IBM Cloud API key using: https://cloud.ibm.com/iam#/apikeys	String		Yes	Yes
   create_cluster	For creating new IBM Kubernetes cluster	String	false	Yes	No
   cluster_name	Name of IBM Kubernetes cluster	String		Yes	No
   event_stream_name	Name of source Event Stream instance to connect to	String		Yes	No
   event_stream_region	Region of the source Event Stream instance (us-south, eu-de, etc.)	String	us-south	Yes	No
   event_stream_resource_group	Resource group name of the source Event Stream instance	String	Default	Yes	No
   event_stream_topic	Name of source Event Stream topic	String		Yes	No
   splunk_index	A repository for Splunk data	String		Yes	No
   splunk_hec_uri	URL for the Splunk - HTTP Event Collector	String		Yes	No
   splunk_hec_token	Auth token for the Splunk - HTTP Event Collector	String		Yes	No
   splunk_connector_name	Name for the splunk connector(Must not already exist)	String		Yes	No

4. Click the "Save Changes".

5. Click "Generate Plan". And, view the progress in the Activities page.

6. Click "Apply Plan". And, view the progress in the Activities page.

On successful deployment,

• You will see the following log messages in Schematic workspace logs.

  

• You will see the following additional service instances in the Resource List page.

  • Schematics workspaces : workspace_name

Access permissions

You must provide the following minimum permissions, for any delegated user, to successfully deploy the Export to Event Streams capability using the Terraform-based automation.

Cloud Services	Resource Group	Permission
Schematics	workspace_resource_group	Service role: Manager
IBM Kubernetes Service	event_stream_resource_group	Platform role : Operator Service role: Manager
Event Streams Topic	event_stream_resource_group	Service role: Writer

Remove Event Streams connector for Splunk using IBM Cloud Schematics

1. Open IBM Cloud Schematics workspace
2. In the workspace list, select the workspace_name previously entered for this Event Streams connector for Splunk capability, and press Delete
3. In the "Delete Workspace" pop-up window,
   • Select 'Delete workspace'
   • Select 'Delete all associated resources'
   • Type the and press the Delete button

Image License Details

Software	URL	Version	License	Provider
Base image from Event Streams	ibmcom/eventstreams-kafkaconnect	2019.2.1-3a2f93e	Apache License Version 2.0	IBM
ibmjava	ibmjava	8-jre		IBM
Kafka	kafka download	2.2.0	Apache License Version 2.0	Apache
kafka-connect-splunk	Kafka Connect for Splunk	v2.0.2	Apache License Version 2.0	Splunk
Kafka Connect Splunk Transformer	Splunk Transformer	v0.0.4	Apache License Version 2.0	IBM

---