Inputs and Preprocessing

SOF-ELK ingests a variety of file formats, which are detailed below. In general, simply placing the files into the appropriate directory is all that's needed. However, if any specific instructions are required for exporting or generating the files, or any preprocessing is needed, the details are provided below.

• SOF-ELK ingests files from the following filesystem locations:
  • /logstash/aws/: JSON-formatted Amazon Web Services CloudTrail log files. Use the included aws-cloudtrail2sof-elk.py loader script.
  • /logstash/azure/: JSON-formatted Microsoft Azure logs. At this time, the following log types are supported: Event Logs, Sign In Logs, Audit Logs, Admin Activity Logs, and Storage Logs.
  • /logstash/gcp/: JSON-formatted Google Compute Platform logs. See the Cloud Evidence Acquisition -> Google Compute Platform (GCP) page for more specific details.
  • /logstash/gws/: JSON-formatted Google Workspace logs extracted using the Google Workspace API.
  • /logstash/hayabusa/: Output from Yamato Security's Hayabusa Windows event log fast forensics timeline generator and threat hunting tool. JSON or JSONL formatted output is supported, as well as CSV output created with the standard profile. Files must be named *.json, *.jsonl, or *.csv, respectively.
  • /logstash/httpd/: Apache logs in common, combined, or vhost-combined formats
  • /logstash/kape/: JSON-format files generated by the KAPE triage collection tool. (See this document for details on which specific output files are currently supported and their required file naming structure.)
  • /logstash/kubernetes/: Kubernetes log files.
  • /logstash/microsoft365/: JSON-formatted Microsoft 365 logs only.
  • /logstash/nfarch/: Archived NetFlow output, formatted as described below.
  • /logstash/passivedns/: Logs from the passivedns utility.
  • /logstash/plaso/: CSV bodyfile-format files generated by the Plaso tool from the log2timeline framework. (See this document for details on creating CSV files in a supported format.)
  • /logstash/syslog/: Syslog-formatted data
    • NOTICE: Remember that syslog DOES NOT reflect the year of a log entry! Therefore, Logstash has been configured to look for a year value in the path to a file. For example: /logstash/syslog/2015/var/log/messages will assign all entries from that file to the year 2015. If no year is present, the current year will be assumed. This is enabled only for the /logstash/syslog/ directory.
  • /logstash/zeek/: JSON-formatted logs from the Zeek Network Security Monitoring platform. These must be in decompressed form. The following Zeek logs are supported:
    • conn.log: Treated like NetFlow and stored in the netflow-* indices.
    • dns.log: Treated like other DNS logs and stored in the logstash-* indices.
    • http.log: Treated like other HTTP logs and stored in the httpdlog-* indices.
    • The following logs are stored in the zeek-* indices:
      • files.log
      • ftp.log
      • notice.log
      • ssl.log
      • weird.log
      • x509.log