Salesforce Event Log File on ELK Stack on Docker
Salesforce Event Log File (ELF) exposes organizational usage trends and user behavior via the EventLogFile sObject. It's a fantastic feed of data that enables customers to derive insights from analytic platforms like ELK.
The ELK stack is an open-source, scalable log management stack that supports exploration, analysis, and visualization of log data.
It consists of:
- Elasticsearch: A Lucene-based search server for storing log data.
- Logstash: ETL process for retrieving, transforming, and pushing logs into data warehouses.
- Kibana: Web GUI for exploring, analyzing, and visualizing log data in Elasticsearch.
Because ELK is on-premise, we are using Docker as a container of settings to simplify the deployment and running of the ELK stack on commoditized hardware.
The Salesforce Event Log File Ruby Gem plug-in simplifies the integration between the ELK stack and Event Log Files by allowing you to easily download and index Salesforce event data every day without a custom integration.
To get started, all you have to do is download this repo, modify a config file, and start the service.
Supported Operating Systems
Linux and Mac OS X.
Windows is not tested and not supported at the moment. However, it reportedly works when running Docker without Makefile.
Install the latest version of Docker from here. You may need to restart the service after installing before you can test it with the hello-world container (e.g. sudo service docker start).
Varies by the amount of event log file data your Salesforce organization generates. It is recommended that you have at least 4GB RAM and 16GB of free hard disk space at the minimum.
Event Log File Logstash Plugin Configuration
This section guides you in setting up Logstash Event Log File plugin configuration. The configuration is defined in the
sfdc_elf.config file. Complete the following parameters in your config file:
- username: Your Salesforce username.
- password: Your Salesforce password.
- client_id: Salesforce connected App client ID. See 'Setting up Salesforce Connected App' section below.
- client_secret: Salesforce connected App client secret. See 'Setting up Salesforce Connected App' section below.
- security_token: Your account's security token only needed when your Force.com organization requires it.
- host: defaults to "login.salesforce.com", Use "test.salesforce.com" for connecting to Sandbox instance.
Running ELF on ELK on Docker
- Ensure you already have Docker installed. Additionally, you need Docker Machine (not boot2docker) if you are using Max OS X.
- Clone or download this GIT repository.
- Complete the Logstash configuration (
sfdc_elf.configfile). See previous section for details.
- We provide Makefile to help you easily manage Docker images. Run your Docker container using
make run. You will need to use your terminal to first navigate to the directory where this GIT repo was downloaded to (i.e. cd ~/Documents/elf_elk_docker-master) before running
make run. Additional information about managing Docker containers:
make removeremoves an already running Docker container.
make resumeresumes an existing but stopped Docker container.
See the image below for interacting with Docker container with
maketool. If you are an advanced Docker user or use Windows platform, you might want to look inside
scripts/directory on how to run (
scripts/run.sh), resume (
scripts/resume.sh), or remove (
scripts/remove.sh) Docker images.
Exploring, Analyzing, and Visualizing Data in Kibana
- Kibana is configured to run on port 8081. You can access it from your web browser using
localhost:8081in Linux. For Max OS X, run
docker-machine ip defaultto determine your virtual machine's IP. Then access Kibana by going to that IP at port 8081 (e.g. http://192.168.98.100:8081).
- Set index pattern in Settings > Indices to
logstash-*and click Create. Some mapping conflicts may occur which is fine. See screenshot below.
- Click Discover and start exploring and visualizing your data.
Report bugs and issues here.
(Listed in no particular order)
- Mohammed Islam
- Abhishek Sreenivasa
- Adam Torman
- A special thanks to all other committers who have contributed to this effort.
Setting up a Salesforce Connected App
Detailed instructions for setting up a Connected App can be found here. When configuring the connected application, ensure the following options are configured:
- Enable OAuth Settings is checked.
- Access and manage data (api) and Access your basic information (id, profile, email, address, phone) are included in your Selected OAuth Scopes.