Malcolm v24.05.0

[mmguero]

Malcolm v24.05.0 contains new features, improvements, bug fixes and component version updates.

v24.04.0...v24.05.0

• Features and enhancements
  • Added ARM64/AArch64 support. Malcolm can now run natively on ARM64 hardware. The ./scripts/configure script should detect the architecture and automatically adjust the image: names in the docker-compose.yml files in Docker deployments, or this can be changed manually by appending -arm64 to the tag for Malcolm's Docker images, e.g., ghcr.io/idaholab/malcolm/zeek:24.05.0-arm64. (Using MISP and STIX/TAXII Intelligence Feeds idaholab/Malcolm#369)
  • Support for new environment variables added to Hedgehog Linux's control_vars.conf for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in the arkime-live container in Malcolm. (discrepancy between environment variables used to populate Arkime's config.ini between Malcolm and Hedgehog idaholab/Malcolm#476)
  • Tweaked some of the default resource-related live capture settings for Suricata and Arkime.
  • Reworked the environment variables used for tuning Zeek live capture resource and performance on both Malcolm and Hedgehog Linux. An in-depth discussion of these tuning parameters can be found in the documentation. (streamline configuration of Zeek live capture worker load balancing using AF_PACKET and fanout idaholab/Malcolm#475)
  • Allow setting the spiDataMaxIndexes variable for Arkime's config.ini file via the ARKIME_SPI_DATA_MAX_INDICES environment variable. (allow setting spiDataMaxIndices for Arkime's config.ini idaholab/Malcolm#471)
  • Allow custom tags to be specified at the point of log file ingestion (i.e., FileBeat) on Malcolm and Hedgehog Linux. This makes it easier to specify custom tags used to group network traffic by sensor. (allow custom tags to be specified at the point of log file ingestion (ie., filebeat) idaholab/Malcolm#463)
  • Handle invalid URLs made to the Malcolm web-based UIs better (with a custom 404/502 page). (handle invalid URLs better (with a 404 instead of a 500) idaholab/Malcolm#461)
  • Switched to official .deb packages for Arkime rather than building from source, reducing build times significantly. (Thanks @awick.)
• Component version updates
  • Suricata to v7.0.5
    • Also, going forward Malcolm will track the latest Suricata release (from the Debian Stable Backports APT repository) rather than what's in the Debian Stable APT repository. (Track latest Suricata release rather than what's in Debian stable APT repository idaholab/Malcolm#462)
  • Arkime to v5.2.0
  • OpenSearch and OpenSearch Dashboards to v2.14.0
  • YARA to v4.5.1
  • Beats to v8.13.4
  • Logstash to v8.13.4
  • YQ to v4.44.1
  • Zeek to v6.2.1
  • Fluent Bit to v3.0.6
  • requests Python library to v2.32.0 for CVE-2024-35195
  • flask-cors Python library on Hedgehog Linux to v4.0.1 for CVE-2024-1681
  • Jinja Python library on Hedgehog Linux to v3.1.4 for CVE-2024-34064
  • Werkzeug Python library on Hedgehog linux to v3.0.3 for CVE-2024-34069
• Bug fixes
  • The code that cleans up already-processed Zeek and Suricata logs after a defined period of time was out of date for the current FileBeat registry behavior and would potentially leave log files around longer than they needed to be. This has been remedied. (clean-processed-folder.py out of date for current filebeat registry behavior idaholab/Malcolm#479)
  • Fixed issue where the BPF capture filter was not passed to Zeek correctly. (BPF capture filter does not seem to be passed to Zeek correctly idaholab/Malcolm#474)
  • The process which queries threat intelligence feeds and generates the corresponding Zeek intel files will no longer relpace existing intel definitions unless it succeeds in pulling definitions from at least one of the specified feeds. (don't overwrite Zeek threat intel files if all sources associated with a type fail idaholab/Malcolm#472)
  • Fixed calculation of memory and CPU resources used in ./scripts/status for Kubernetes deployment. (error reporting memory and CPU resources in scripts/malcolm_kubernetes.py idaholab/Malcolm#467)
• Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • Malcolm
    • Added ARKIME_SPI_DATA_MAX_INDICES to arkime.env with a default value of 7, which manifests as spiDataMaxIndexes in Arkime's config.ini. If you are changing the Arkime index period from daily to weekly, hourly, etc., you may wish to adjust this value. (allow setting spiDataMaxIndices for Arkime's config.ini idaholab/Malcolm#471)
    • Added EXTRA_TAGS to upload-common.env for specifying custom tags to be associated with logs forwarded to Logstash by FileBeat. (allow custom tags to be specified at the point of log file ingestion (ie., filebeat) idaholab/Malcolm#463)
    • A number of new and modified environment variables are available and can be added to zeek-live.env for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (streamline configuration of Zeek live capture worker load balancing using AF_PACKET and fanout idaholab/Malcolm#475)
  • Hedgehog Linux
    • A number of new and modified environment variables are available for control_vars.conf for tuning the resource utilization and performance of Zeek's live capture, see the documentation for details. (streamline configuration of Zeek live capture worker load balancing using AF_PACKET and fanout idaholab/Malcolm#475)
    • Added support for new environment variables added to Hedgehog Linux's control_vars.conf for tuning capture-related settings for Arkime on the sensor, bringing them into parity with those that were available in the arkime-live container in Malcolm. (discrepancy between environment variables used to populate Arkime's config.ini between Malcolm and Hedgehog idaholab/Malcolm#476)