You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We need to look at adding to suricata_config_populate.py the ability to generate suricata.yaml such that suricata's eve.json is split/rotated so it doesn't grow as large.
There are a few things we could do here as possibilities
split out logs by protocol (e.g., see the types in the example below)
# Configure the type of alert (and other) logging you would like.outputs:
# a line based alerts log similar to Snort's fast.log
- fast:
enabled: nofilename: fast.logappend: yes#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'# Extensible Event Format (nicknamed EVE) event log in JSON format
- eve-log:
enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve-alerts.jsoncommunity-id: truecommunity-id-seed: 0types:
- alert:
metadata: yes # enable inclusion of app layer metadata with alert. Default yestagged-packets: yes# Suricata HTTP Logs
- eve-log:
enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve-http-%Y-%m-%d-%H.jsonrotate-interval: hourcommunity-id: truecommunity-id-seed: 0types:
- http:
extended: yes# Suricata DNS Logs
- eve-log:
enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve-dns-%Y-%m-%d-%H.jsonrotate-interval: hourcommunity-id: truecommunity-id-seed: 0types:
- dns:
version: 2enabled: yesrequests: yesresponse: yesformats: [detailed, grouped]types: [a, aaaa, cname, mx, ns, ptr, txt]# Suricata TLS Logs
- eve-log:
enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve-tls-%Y-%m-%d-%H.jsonrotate-interval: hourcommunity-id: truecommunity-id-seed: 0types:
- tls:
extended: yessession-resumption: nocustom: [subject, issuer, session_resumed, serial, fingerprint, sni, version, not_before, not_after, certificate, ja3, ja3s]# Suricata Flow Logs
- eve-log:
enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve-flow-%Y-%m-%d-%H.jsonrotate-interval: hourcommunity-id: truecommunity-id-seed: 0types:
- flow:
# Suricata Other (dnp3, ftp, nfs, smb, ssh.....) Logs
- eve-log:
enabled: yesfiletype: regular #regular|syslog|unix_dgram|unix_stream|redisfilename: eve-other-%Y-%m-%d-%H.jsonrotate-interval: hourcommunity-id: truecommunity-id-seed: 0types:
- smtp:
extended: yescustom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]md5: [body, subject]
- files:
force-magic: noforce-hash: [mdt]
- dnp3
- ftp
- nfs
- smb
- tftp
- ikev2
- krb5
- snmp
- sip
- ssh
- dhcp:
enabled: yesextended: no# a line based log of HTTP requests (no alerts)
- http-log:
enabled: nofilename: http.logappend: yes#extended: yes # enable this for extended logging information#custom: yes # enable the custom logging format (defined by customformat)#customformat: "%{%D-%H:%M:%S}t.%z %{X-Forwarded-For}i %H %m %h %u %s %B %a:%p -> %A:%P"#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'# a line based log of TLS handshake parameters (no alerts)
- tls-log:
enabled: no # Log TLS connections.filename: tls.log # File to store TLS logs.append: yes#extended: yes # Log extended information like fingerprint#custom: yes # enabled the custom logging format (defined by customformat)#customformat: "%{%D-%H:%M:%S}t.%z %a:%p -> %A:%P %v %n %d %D"#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'# output TLS transaction where the session is resumed using a# session id#session-resumption: no# output module to store certificates chain to disk
- tls-store:
enabled: no#certs-log-dir: certs # directory to store the certificates files# Packet log... log packets in pcap format. 3 modes of operation: "normal"# "multi" and "sguil".## In normal mode a pcap file "filename" is created in the default-log-dir,# or as specified by "dir".# In multi mode, a file is created per thread. This will perform much# better, but will create multiple files where 'normal' would create one.# In multi mode the filename takes a few special variables:# - %n -- thread number# - %i -- thread id# - %t -- timestamp (secs or secs.usecs based on 'ts-format'# E.g. filename: pcap.%n.%t## Note that it's possible to use directories, but the directories are not# created by Suricata. E.g. filename: pcaps/%n/log.%s will log into the# per thread directory.## Also note that the limit and max-files settings are enforced per thread.# So the size limit when using 8 threads with 1000mb files and 2000 files# is: 8*1000*2000 ~ 16TiB.## In Sguil mode "dir" indicates the base directory. In this base dir the# pcaps are created in the directory structure Sguil expects:## $sguil-base-dir/YYYY-MM-DD/$filename.<timestamp>## By default all packets are logged except:# - TCP streams beyond stream.reassembly.depth# - encrypted streams after the key exchange#
- pcap-log:
enabled: {{ .Values.enable_conditional_pcap }}filename: sig-%n-%t.pcap# File size limit. Can be specified in kb, mb, gb. Just a number# is parsed as bytes.limit: 5mb# If set to a value, ring buffer mode is enabled. Will keep maximum of# "max-files" of size "limit"max-files: 2000# Compression algorithm for pcap files. Possible values: none, lz4.# Enabling compression is incompatible with the sguil mode. Note also# that on Windows, enabling compression will *increase* disk I/O.compression: none# Further options for lz4 compression. The compression level can be set# to a value between 0 and 16, where higher values result in higher# compression.#lz4-checksum: no#lz4-level: 0mode: multi # normal, multi or sguil.# Directory to place pcap files. If not provided the default log# directory will be used. Required for "sguil" mode.dir: /var/log/suricata/pcaps#ts-format: usec # sec or usec second format (default) is filename.sec usec is filename.sec.usecuse-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packetshonor-pass-rules: no # If set to "yes", flows in which a pass rule matched will stop being logged.# Use "all" to log all packets or use "alerts" to log only alerted packets and flows or "tag"# to log only flow tagged via the "tag" keywordconditional: alerts# a full alert log containing much information for signature writers# or for investigating suspected false positives.
- alert-debug:
enabled: nofilename: alert-debug.logappend: yes#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'# alert output to prelude (https://www.prelude-siem.org/) only# available if Suricata has been compiled with --enable-prelude
- alert-prelude:
enabled: noprofile: suricatalog-packet-content: nolog-packet-header: yes# Stats.log contains data from various counters of the Suricata engine.
- stats:
enabled: yesfilename: stats.logappend: yes # append to file (yes) or overwrite it (no)totals: yes # stats for all threads merged togetherthreads: no # per thread stats#null-values: yes # print counters that have value 0. Default: no
The text was updated successfully, but these errors were encountered:
Added new functionality controlled by environment to suricata_config_populate.py that take effect only for live capture (i.e., when SURICATA_LIVE_CAPTURE is true):
SURICATA_EVE_FILENAME_PATTERN - how live eve.json files are named (default eve-%Y%m%d_%H%M%S.json)
Removed the script eve-clean-logs.sh from the Suricata containers for cleaning old eve.json logs. This is now handled by the filebeat container by the clean-processed-folder.py script in the same way the Zeek logs are, with behavior controlled by LOG_CLEANUP_MINUTES env variable for Suricata logs.
Moved the LOG_CLEANUP_MINUTES and ZIP_CLEANUP_MINUTES from upload-common.env to filebeat.env as filebeat is the only container that uses those.
Adjusted the filebeat.yml configuration files on Malcolm and hedgehog to look for eve*.json as the matching wildcard for harvesting.
Renamed the script filebeat-clean-zeeklogs-processed-folder.py to clean-processed-folder.py as it is not used for more than just Zeek scripts. Added logic in there differentiating between Zeek and Suricata logs, and check both the suricata and suricata-live folders for old logs to prune. As the suricata documentation reads,
In addition to the SIGHUP style rotation discussed above, some outputs support their own time and date based rotation, however removal of old log files is still the responsibility of external tools. These outputs include: Eve
mmguero
added a commit
to mmguero-dev/Malcolm
that referenced
this issue
Mar 18, 2024
We need to look at adding to suricata_config_populate.py the ability to generate suricata.yaml such that suricata's eve.json is split/rotated so it doesn't grow as large.
There are a few things we could do here as possibilities
rotate-interval: hour
)types
in the example below)The text was updated successfully, but these errors were encountered: