DShield Sensor Log Collection with Elasticsearch

Introduction

This is fork and a significant update from the initial publication on the ISC Storm Center website by Scott Jensen as a BACS paper and the scripts published in Github.
https://github.com/fkadriver/Dshield-ELK
https://isc.sans.edu/diary/DShield+Sensor+Monitoring+with+a+Docker+ELK+Stack+Guest+Diary/30118

What it is Used For

This docker is custom built to be used with the DShield Honeypot to collect and parse the logs and collect its data in a visual and easy to search for research purposes. The suggested installation is to install the DShield sensor in a Rasperry using PI Raspbian OS or a system running Ubuntu 20.04 LTS either in your network or in the cloud of your choice.

• This was tested on Ubuntu 20.04 LTS
  
• Step 1 build Ubuntu
  
• Step 2 install docker and ELK
  
• Step 3 install and configure Filebeat on DShield Sensor(s)
  

Ubuntu Setup

• Ubuntu 20.04 LTS Live Server 64-Bit
  
• Minimum 8+ GB RAM
  
  • If the amount of RAM assigned to each containers (see below) is more than 2GB, consider increasing the server RAM capacity.
    
• 4-8 Cores
  
• Minimum 40 GB partition assigned to /var/lib/docker
  
• Adding a 40 GB to a VM: https://github.com/bruneaug/DShield-SIEM/blob/main/build_40gb_docker_partition.txt

Elastic Packages Installed

ELK Current Version: 8.13.0 (Updated March 2024)

• Kibana
• Elasticsearch
• Logstash
• Elastic-Agent

Note: To update ELK server components, follow these steps:
https://github.com/bruneaug/DShield-SIEM/blob/main/README.md#download-github-update

Install docker

$ sudo apt-get install ca-certificates curl gnupg network-manager txt2html
$ sudo install -m 0755 -d /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ sudo chmod a+r /etc/apt/keyrings/docker.gpg
$ echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt update && sudo apt upgrade
$ sudo reboot (if update were applied)
$ sudo apt-get install -y jq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin pip
$ sudo systemctl enable docker

Configure and install DShield ELK

$ git clone https://github.com/bruneaug/DShield-SIEM.git
$ chmod 754 ~/DShield-SIEM/scripts/cowrie-setup.sh
$ mkdir scripts
$ mv DShield-SIEM/AddOnScripts/startelk.sh scripts/
$ mv DShield-SIEM/AddOnScripts/parsing_tty.sh scripts
$ mv DShield-SIEM/AddOnScripts/rename_arkime_pcap.sh scripts
$ chmod 754 scripts/*.sh

The script ~/scripts/startelk.sh should be edited if you modified the elastic password to reflect your user account password. Default is currently student.
The parsing_tty.sh script will be configured later in another document.
$ cd ~/DShield-SIEM

Note: Before installation, you can edit the .env file to make any derided changes.
- Current default password for elastic is student
Memory Limits in .env are the most memory that docker will allocate for each of the ELK containers.
Default to 2147483648 (2GB) but can be expanded if you have the resources

• The default name servers in Logstash are set in the following files to: nameserver => [ "8.8.8.8", "9.9.9.9" ]
• If you want to change the default nameserver(s), go to the following directory for the files
  • $ cd logstash/pipeline
  • logstash-200-filter-cowrie.conf
    
  • logstash-201-filter-iptables.conf
    
  • logstash-202-filter-cowrie-webhoneypot.conf
    You can keep these default or edit each files and change them.

$ sudo docker compose up -d (For setup or any changes)

Installation Completed

Access Kibana Interface

Web Access: http://serverIP:5601

Configure Management -> Stack Monitoring

• Select "Or, set up with self monitoring"
  
• Monitoring is currently off -> Select: Turn on monitoring
  

• Login Kibana with username: elastic and default password if it hasn't been changed: student
  

Configuring elastic-agent

The elastic-agent will be used to ingest threat intelligence. It can also be used to do other things that won’t be covered here.

• From the dropdown menu, select Management → Fleet →Settings → Edit Outputs (Actions)
  

• Login server via SSH
  

• Copy ca.crt certificate to /tmp
  $ sudo docker cp es01:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.

• Get a copy of Elasticsearch CA trusted fingerprint
  $ sudo openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g

• The output will look like this:
  673FB617E15CCCE73F9B647EF99449642A19CFC1D75BF5772047DA99DB950844

• Get Content of Elasticsearch CA Certificate to Apply to Advanced YAML configuration. Type the command because it doesn't copy well
  $  sudo cat /tmp/ca.crt | sed -r 's/(.*)/ \1/g'

Format must be exactly like this. Copy the output of the certificate in Notepad or Notepad++ and format exactly like this.
It needs 2 spaces before certificate_authorities: and the dash (-) and it needs 4 spaces from the pipe (|) all the way down to the end of -----END CERTIFICATE-----
sed will add the 4 spaces with the previous command against the CA certificate

• Change the Hosts to: https://es01:9200

Save and apply settings after making the changes and adding the certificate information. Followed by Save and deploy

The raw output for the Certificate should look like this with the same spaces as per this picture. Two spaces from the left for the certificate certificate_authorities: and four spaces from the left from the pipe (|) to the end of certificate [9]:

Under Settings, configure the Fleet server hosts by either edit or Add Fleet Server configured as follows:

Next phase is to Select Agent Policy → Add Agent → Enroll in Fleet → Add Fleet Server

• Provide a Name: es01
• Provide URL: https://fleet-server:8220
• Last: Generate Fleet Server policy
• Select: RPM
• Copy starting at: elastic-agent enroll \ to the end of …port=8220

We are going to need this information to setup our fleet server.
Login via SSH to the fleet-server and make sure the fleet-server is running before setting up our agent:

$ sudo docker start fleet-server
$ sudo docker exec -ti fleet-server bash
$ ./elastic-agent status (check it is running)

This is an example of what need to be copied to the fleet server. Ensure the fleet server es is: https://es01:9200
Add the bold section after port=8220 because are certificates are self-generated. This will ensure the agent takes the update.

The token and fingerprint will be different than this example but what is in italic and bolded must be added for the certificat to load:

elastic-agent enroll \
--url=https://fleet-server:8220 \
--fleet-server-es=https://es01:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MDU0NDg3MDMwNTI6NkNxcWlCeTRRVmlhYW0yeldhN3pGZw \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca-trusted-fingerprint=76DA77DAE186F8CFBA9E87D450D5419B68E2555A9BD57795611C0545ED0BF03F \
--fleet-server-port=8220 \
--certificate-authorities=/certs/ca/ca.crt \
--fleet-server-es-ca=/certs/es01/es01.crt \
--insecure

This will replace your current settings. Do you want to continue? [Y/n]: Y

{"log.level":"info","@timestamp":"2024-01-17T00:00:40.404Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":411},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-01-17T00:00:42.774Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":788},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-01-17T00:00:43.073Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":479},"message":"Starting enrollment to URL: https://a4a1ada63084:8220/","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-01-17T00:00:44.152Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":277},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.

From your current location, verify it installed correctly
$ ./elastic-agent status

Now that the Fleet Server is connected, close this Windows and lets proceed to the next step.

In Elastic Management → Fleet, refresh Agents and this is what shows up:

The server is now ready to install Threat Intel Agents to be used in Security (SIEM portion) against the honeypot logs.
The next step is to select Agent policies → Fleet Server Policy → Add integration:
Use this Dashboard to view the metrics collected by elastic-agent:

• Select and Add AlienVault OTX (need an API key)
  
• Select AbuseCH (no API key needed)
  
• Select Threat Intelligence Utilities
  
• Elasticsearch
  
• Select Kibana
  
• Select Docker
  

Fleet Server Policy Example

• In Elastic Management → Installed Integration
  
• Select each of the installed integration, then select Settings and enable the tab to keep the policy up to date:
  

Configuring Security → Rules

• Select Rules → Detection rules (SIEM) → Add Elastic rules
• Under Search Tags: Rule Type: Indicator Match (add at the minimum those 4 rules)
• Install and enable those 4 rules
• You can look through the rules and enable those other rules that you want to try against your honeypot data.
  

Configure Management → Stack Management → Advanced Settings

Find Elasticsearch Indices and add at the end of the list (comma cowrie *):

• ,cowrie*
  
• Save changes for these logs to be analyzed by the SIEM part of ELK.
  The final result should look like this:

Review the Activity Captured by the Rules in Alert trend

Setup Filebeat on DShield Sensor - Logs to ELK

After adding the webhoneypot.sh script, add the Filebeat package to the DShield Sensor to send the logs the Elasticsearch.

If use the following to install the Filebeat package using [3] the following commands:

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
$ echo "deb https://artifacts.elastic.co/packages/oss-8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
$ sudo apt-get update && sudo apt-get install filebeat elastic-agent softflowd

Download the custom filebeat.yml file that will forward the logs the Elasticsearch:

$ sudo curl https://raw.githubusercontent.com/bruneaug/DShield-SIEM/main/filebeat.yml -o /etc/filebeat/filebeat.yml

• Edit the filebeat.yml and change the IP address to the logstash parser (192.168.25.23) to match the IP used by Logstash:
  $ sudo vi /etc/filebeat/filebeat.yml
  

  output.logstash:
  hosts: ["192.168.25.23:5044"]
  

Start Filebeat

$ sudo systemctl enable filebeat
$ sudo systemctl start filebeat
$ sudo systemctl status filebeat
$ sudo systemctl enable elastic-agent
$ sudo systemctl start elastic-agent
$ sudo systemctl enable softflowd
$ sudo systemctl start softflowd

Filebeat Tracking File

Filebeat tracks the events it has processed with a file located called log.json, if deleted, all the events that were previous sent to Elasticsearch will be reprocessed when filebeat is restarted.
The location of this file:
$ cd /var/lib/filebeat/registry/filebeat

Want to add Elastic-Agent to other Devices?

Follow this step-by-step documents to install the elastic-agent to the DShield sensor
https://github.com/bruneaug/DShield-SIEM/blob/main/Configure-Elastic-Agent.pdf

Interface - Logs DShield Sensor Overview

To access the Dashboard select Analytics -> Dashboard -> [Logs DShield Sensor] Overview

Starting ELK after a Reboot

This script will start all the ELK components and if installed, the Arkime services.

Install startelk.tgz tarball to ELK server as follow:
$ cd ~/DShield-SIEM/AddOn $ sudo tar zxvf startelk.tgz -C /

Edit the Script and Update the User Account Password

$ sudo vi /etc/init.d/startelk.sh

The script is configured with the default password to sudo: training
You need to change it to the account's password in use.
You need to change the account under which location DShield-SIEM is located.

PASSWORD="training" ELK="**/home/guy/**DShield-SIEM"

Useful Docker Commands

$ sudo docker compose rm -f -v (clear setup but need to run up -d again)
$ sudo docker compose up -d (reload container with changes)
$ sudo docker compose up --build -d
$ sudo docker compose up --build --force-recreate -d (force a rebuild if the container hasn't changed)
$ sudo docker compose start/stop
$ sudo docker compose ps (list running containers)
$ sudo docker stats (shows status of container)
$ sudo docker container ls/ps
$ sudo docker network ls (network listing)
$ sudo docker stats (shows status of container)
$ sudo docker system prune -a (Remove everything)
$ sudo docker logs kibana (troubleshooting docker)
$ sudo docker compose logs --follow (debugging)
$ sudo docker compose down --remove-orphans && sudo docker compose up --build -d (Removed or renamed orphan container)
$ sudo docker rm -f cowrie (remove a container)

$ sudo docker stop fleet-server
$ sudo docker stop logstash
$ sudo docker inspect logstash

Download Github Update

$ cd DShield-SIEM
$ sudo docker compose stop
$ git pull (Update the code from Github)
$ sudo docker compose rm -f -v
$ sudo docker compose up --build -d

Remove a Container that Fail to Start

$ sudo docker inspect logstash

Login each Container

$ sudo docker exec -ti es01 bash
$ sudo docker exec -ti logstash bash
$ sudo docker exec -ti kibana bash
$ sudo docker exec -ti fleet-server bash
$ sudo docker exec -ti cowrie bash

Copying Files Between Docker & Local User

$ sudo docker cp server:/usr/share/elastic-agent/elastic-agent.yml .
$ sudo docker cp elastic-agent.yml fleet-server:/usr/share/elastic-agent/

Information on Elastic with the Console

These command are run from the Dev Tool -> Console

GET _nodes/http?pretty (Get a list and information of all the nodes)
GET _security/_authenticate

References

[1] https://isc.sans.edu/tools/honeypot/
[2] https://www.elastic.co/downloads/beats/filebeat
[3] https://www.elastic.co/guide/en/beats/filebeat/8.8/setup-repositories.html#_apt
[4] https://isc.sans.edu/diary/DShield+Honeypot+Activity+for+May+2023/29932
[5] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+to+Elasticsearch/29458
[6] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+Analysis/29412
[7] https://github.com/jslagrew/cowrieprocessor/blob/main/submit_vtfiles.py
[8] https://handlers.sans.edu/gbruneau/elastic.htm
[9] https://www.elastic.co/guide/en/fleet/current/secure-connections.html
[10] https://www.docker.elastic.co/