beats

Using Beats to forward host logs to Malcolm

Because Malcolm uses components of the open source data analysis platform Elastic Stack, it can accept various host logs sent from Beats, Elastic Stack's lightweight data shippers. These Beats generally include prebuilt Kibana dashboards for each of their respective data sets.

Examples

Some examples include:

• Auditbeat
  • auditd logs on Linux hosts
  • file integrity monitoring on Linux, macOS (Darwin) and Windows hosts
  • system state including host, process, login, package, socket and user information on Linux, with some data sets supported on macOS and Windows hosts (apparently not available with the Open Source Elastic license)
• Filebeat
  • system logs (syslog and authentication logs) on Linux hosts
  • log output from many products across Beats-supported platforms
  • arbitrary textual log files
• Metricbeat
  • system resource utilization and process information
  • metrics from many products across Beats-supported platforms
• Packetbeat
  • host-based packet inspection for many protocols (supports libpcap on Linux, macOS and Windows; and af_packet on Linux)
• Winlogbeat
• Custom Beats
• Community-contributed Beats

Convenience configuration scripts and sample configurations

Two scripts are provided here for your convenience in configuring and running Beats to forward log data to Malcolm: beat_config.py and beat_run.py. These Python scripts should run on Linux, macOS and Windows hosts with either Python 2 or Python 3.

Sample configurations are also provided for several beats for Linux and Windows hosts, as well as Vagrantfiles for setting up and running VirtualBox VMs under Vagrant.

For further information, downloads, documentation or support for Beats, see the Beats Platform Reference or the Beats category on the Elastic forums.

Example: Windows configuration and run

PS C:\Program Files\winlogbeat> dir

    Directory: C:\Program Files\winlogbeat

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         7/27/2020   8:49 AM                kibana
d-----         7/27/2020   8:49 AM                module
-a----         3/26/2020   5:33 AM             41 .build_hash.txt
-a----         7/27/2020   8:50 AM          25799 beat_common.py
-a----         7/27/2020   8:50 AM           2525 beat_config.py
-a----         7/27/2020   8:50 AM           2244 beat_run.py
-a----         3/26/2020   5:32 AM         163122 fields.yml
-a----         7/27/2020   8:49 AM            878 install-service-winlogbeat.ps1
-a----         3/26/2020   4:44 AM          13675 LICENSE.txt
-a----         3/26/2020   4:44 AM         328580 NOTICE.txt
-a----         3/26/2020   5:33 AM            825 README.md
-a----         3/26/2020   5:33 AM            254 uninstall-service-winlogbeat.ps1
-a----         3/26/2020   5:33 AM       47818752 winlogbeat.exe
-a----         3/26/2020   5:32 AM          47900 winlogbeat.reference.yml
-a----         7/27/2020   8:50 AM           1349 winlogbeat.yml


PS C:\Program Files\winlogbeat> .\beat_config.py -c .\winlogbeat.yml -b winlogbeat

Append connectivity boilerplate to .\winlogbeat.yml? (y/N): y

Created winlogbeat keystore

Configure winlogbeat Elasticsearch connectivity? (Y/n): y

Enter Elasticsearch connection protocol (http or https) [https]: https

Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none

Enter Elasticsearch connection host: 172.15.0.41:9200

Configure winlogbeat Kibana connectivity? (Y/n): y

Enter Kibana connection protocol (http or https) [https]: https

Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none

Enter Kibana connection host: 172.15.0.41:5601

Configure winlogbeat Kibana dashboards? (Y/n): y

Enter directory containing Kibana dashboards [C:\Program Files\winlogbeat\kibana]: C:\Program Files\winlogbeat\kibana

Enter HTTP/HTTPS server username: sensor
Enter password for sensor:
Enter password for sensor (again):

Generated keystore for winlogbeat
BEAT_KIBANA_SSL_VERIFY
BEAT_ES_HOST
BEAT_ES_PROTOCOL
BEAT_ES_SSL_VERIFY
BEAT_KIBANA_HOST
BEAT_HTTP_PASSWORD
BEAT_HTTP_USERNAME
BEAT_KIBANA_DASHBOARDS_ENABLED
BEAT_KIBANA_DASHBOARDS_PATH
BEAT_KIBANA_PROTOCOL

PS C:\Program Files\winlogbeat> .\beat_run.py -c .\winlogbeat.yml -b winlogbeat

2020-07-27T09:00:17.472-0700    INFO    instance/beat.go:622    Home path: [C:\Program Files\winlogbeat] Config path: [C:\Program Files\winlogbeat] Data path: [C:\Program Files\winlogbeat] Logs path: [C:\Program Files\winlogbeat\logs]
2020-07-27T09:00:17.474-0700    INFO    instance/beat.go:630    Beat ID: c38487f0-ea87-477b-aa93-376eb40949f4
…
^C
KeyboardInterrupt
2020-07-27T09:00:24.783-0700    INFO    instance/beat.go:445    winlogbeat stopped.

Example: Linux configuration and run

root@vagrant:/opt/filebeat# ls -l
total 4
-rw------- 1 root root 431 Jul 27 16:08 filebeat.yml

root@vagrant:/opt/filebeat# beat_config.py -c ./filebeat.yml -b filebeat

Append connectivity boilerplate to ./filebeat.yml? (y/N): y 

Create symlink to module path /usr/share/filebeat/module as /opt/filebeat/module? (Y/n): y

Created filebeat keystore

Configure filebeat Elasticsearch connectivity? (Y/n): y

Enter Elasticsearch connection protocol (http or https) [https]: https

Enter Elasticsearch SSL verification (none (for self-signed certificates) or full) [none]: none

Enter Elasticsearch connection host: 172.15.0.41:9200

Configure filebeat Kibana connectivity? (Y/n): y

Enter Kibana connection protocol (http or https) [https]: https

Enter Kibana SSL verification (none (for self-signed certificates) or full) [none]: none

Enter Kibana connection host: 172.15.0.41:5601

Configure filebeat Kibana dashboards? (Y/n): y

Enter directory containing Kibana dashboards [/usr/share/filebeat/kibana]: /usr/share/filebeat/kibana

Enter HTTP/HTTPS server username: sensor
Enter password for sensor: 
Enter password for sensor (again): 

Generated keystore for filebeat
BEAT_KIBANA_PROTOCOL
BEAT_KIBANA_SSL_VERIFY
BEAT_ES_PROTOCOL
BEAT_ES_SSL_VERIFY
BEAT_KIBANA_DASHBOARDS_ENABLED
BEAT_KIBANA_DASHBOARDS_PATH
BEAT_ES_HOST
BEAT_HTTP_PASSWORD
BEAT_HTTP_USERNAME
BEAT_KIBANA_HOST

root@vagrant:/opt/filebeat# beat_run.py -c ./filebeat.yml -b filebeat

2020-07-27T16:12:43.270Z    INFO    instance/beat.go:622    Home path: [/opt/filebeat] Config path: [/opt/filebeat] Data path: [/opt/filebeat/data] Logs path: [/opt/filebeat/logs]
2020-07-27T16:12:43.270Z    INFO    instance/beat.go:630    Beat ID: 759019e0-705c-4a16-87a2-52e9a5f6e799
…
^C
KeyboardInterrupt
2020-07-27T16:13:10.816Z INFO    beater/filebeat.go:443  Stopping filebeat

Copyright

Malcolm is Copyright 2020 Battelle Energy Alliance, LLC, and is developed and released through the cooperation of the Cybersecurity and Infrastructure Security Agency of the U.S. Department of Homeland Security.

See License.txt for the terms of its release.

Contact information of author(s):

Seth Grover