Skip to content

VAST 2021.04.29
Compare
Choose a tag to compare
@dominiklohmann dominiklohmann released this 29 Apr 08:45
2021.04.29
f7d96f8
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

We’re happy to announce our releases 2021.04.29 of VAST. This is a maintenance release from the user side and developers can now enjoy a mature plugin framework.

Our community chat moved to Gitter. Find it at gitter.im/tenzir/vast. You can also join via Matrix at #tenzir_vast:gitter.im. The main driver for our decision to move was that our EMS-hosted server results in a confusing workflow where interested users have to create a matrix account on a third-party homeserver before they can participate in our chat. Gitter makes it easy to participate for everybody and still runs on top of Matrix, so we get the best of both worlds now.

Maturing VAST’s Plugin Framework

Plugin developers have a whole new bag of toys to choose from. In the past month, we’ve added three new plugin types: Component Plugins are a very general plugin type that runs within the VAST node and has access to the VAST node. Internally, Analyzer Plugins are now built on-top of Component Plugins. We plan to do more with them in the near future, so stay tuned for what’s to come. Additionally, we’ve added Reader Plugins and Writer Plugins, which allow for adding new import and export formats easily, especially when they depend on external libraries. As a starting point, we have converted our PCAP import and export formats into a new PCAP plugin.

Custom Disk Monitor Checks

The Disk Monitor now supports custom check conditions. Users can now specify a vast.start.disk-budget-check-binary option that contains an executable. This can be useful in cases where stat() does not give the correct answer, e.g., on compressed filesystems, or to achieve a more flexible deployment.

A customer of ours achieved a more flexible deployment by configuring the disk monitor to use disk usage percentages instead of absolute directory sizes. Here’s how they’ve done it:

#! /bin/sh

# File: disk-monitor-du.sh
# Print percentage of the disk used by the VAST database.

df --output=pcent "$1" | tr -dc '0-9'
# File: /etc/vast/vast.yaml

vast:
  start: 
    # Trigger removal of old data when database size exceeds 95%
    # of available disk until size is below 90%.
    disk-budget-check-binary: ‘/path/to/disk-monitor-du.sh’
    disk-budget-high: 95
    disk-budget-low: 90

Smaller Things

We removed some previously deprecated options. Please take a look at the breaking changes section in the changelog for more information.

ISO8601-formatted dates with a non-UTC timezone were incorrectly adjusted to the UTC timezone. This has been corrected, all newly ingested data will have correct timezone adjustments. Within VAST, all timestamps are stored in the UTC timezone.

The performance of worst-case export queries improved considerably. A query that selects every second event from the database now completes up to 5 times faster.

Acknowledgements

We want to thank our open-source community for numerous contributions and interactions. This month, we want to thank in particular:

  • @satta contributed schema improvements to the Suricata schema. We also upstreamed his Debian patches to VAST.
  • @norg again provided invaluable feedback from an ops-perspective, and tested the new Disk Monitor features with us.
  • Code Intelligence found a crash by fuzzing our command-line parser when encountering a flag with a missing value in the last position of a command invocation. We fixed this issue.

Changelog Highlights

As always, you can find the full scoop in our changelog.

⚡ Breaking Changes

  • The previously deprecated (#1409) option vast.no-default-schema no longer exists. #1507

  • Plugins configured via vast.plugins in the configuration file can now be specified using either the plugin name or the full path to the shared plugin library. We no longer allow omitting the extension from specified plugin files, and recommend using the plugin name as a more portable solution, e.g., example over libexample and /path/to/libexample.so over /path/to/libexample. #1527

  • The previously deprecated usage (#1354) of format-independent options after the format in commands is now no longer possible. This affects the options listen, read, schema, schema-file, type, and uds for import commands and the write and uds options for export commands. #1529

  • Plugins must define a separate entrypoint in their build scaffolding using the argument ENTRYPOINT to the CMake function VASTRegisterPlugin. If only a single value is given to the argument SOURCES, it is interpreted as the ENTRYPOINT automatically. #1549

  • To avoid confusion between the PCAP plugin and libpcap, which both have a library file named libpcap.so, we now generally prefix the plugin library output names with vast-plugin-. E.g., The PCAP plugin library file is now named libvast-plugin-pcap.so. Plugins specified with a full path in the configuration under vast.plugins must be adapted accordingly. #1593

⚠️ Changes

  • The metrics for Suricata Eve JSON and Zeek Streaming JSON imports are now under the categories suricata-reader and zeek-reader respectively so they can be distinguished from the regular JSON import, which is still under json-reader. #1498

  • VAST now ships with a schema record type for Suricata's rfb event type. #1499 @satta

  • The exporter.hits metric has been removed. #1514 #1574

  • We upstreamed the Debian patches provided by @satta. VAST now prefers an installed tsl-robin-map>=0.6.2 to the bundled one unless configured with --with-bundled-robin-map, and we provide a manpage for lsvast if pandoc is installed. #1515

  • The Suricata dns schema type now defines the dns.grouped.A field containing a list of all returned addresses. #1531

  • The status output of Analyzer Plugins moved from the importer.analyzers key into the top-level record. #1544

  • The new option --disable-default-config-dirs disables the loading of user and system configuration, schema, and plugin directories. We use this option internally when running integration tests. #1557

  • Building VAST now requires CMake >= 3.15. #1559

🎁 Features

  • The disk monitor gained a new vast.start.disk-budget-check-binary option that can be used to specify an external binary to determine the size of the database directory. This can be useful in cases where stat() does not give the correct answer, e.g. on compressed filesystems. #1453

  • It is now possible to build plugins against an installed VAST. This requires a slight adaptation to every plugin's build scaffolding. The example plugin was updated accordingly. #1532

  • Component Plugins are a new category of plugins that execute code within the VAST server process. Analyzer Plugins are now a specialization of Component Plugins, and their API remains unchanged. #1544 #1547 #1588

  • Reader Plugins and Writer Plugins are a new family of plugins that add import/export formats. The previously optional PCAP format moved into a dedicated plugin. Configure with --with-pcap-plugin and add pcap to vast.plugins to enable the PCAP plugin. #1549

🪲 Bug Fixes

  • VAST no longer refuses to start when any of the configuration file directories is unreadable, e.g., because VAST is running in a sandbox. #1533

  • The CSV reader no longer crashes when encountering nested type aliases. #1534

  • The command-line parser no longer crashes when encountering a flag with missing value in the last position of a command invocation. #1536

  • A bug in the parsing of ISO8601 formatted dates that incorrectly adjusted the time to the UTC timezone has been fixed. #1537

  • The shutdown logic contained a bug that would make the node fail to terminate in case a plugin actor is registered at said node. #1563

  • A race condition in the shutdown logic that caused an assertion was fixed. #1563

  • The exporter.selectivity metric is now 1.0 instead of NaN for idle periods. #1574

  • VAST no longer renders JSON numbers with non-finite numbers as NaN, -NaN, inf, or -inf, resulting in invalid JSON output. Instead, such numbers are now rendered as null. #1574

  • Specifying relative CMAKE_INSTALL_*DIR in the build configuration no longer causes VAST not to pick up system-wide installed configuration files, schemas, and plugins. The configured install prefix is now used correctly. The defunct VAST_SYSCONFDIR, VAST_DATADIR, and VAST_LIBDIR CMake options no longer exist. Use a combination of CMAKE_INSTALL_PREFIX and CMAKE_INSTALL_*DIR instead. #1580

  • Spaces before SI prefixes in command line arguments and configuration options are now generally ignored, e.g., it is now possible to set the disk monitor budgets to 2 GiB rather than 2GiB. #1590

Assets 8