Skip to content

VAST 2021.06.24

Compare
Choose a tag to compare
@dominiklohmann dominiklohmann released this 24 Jun 09:38
a2998ed

We’re happy to announce our release 2021.06.24 of VAST. This month we focused primarily on maintenance and streamlining of existing functionality.

Dependency Changes

Apache Arrow is now a required VAST dependency. Transforms already leverage Arrow for efficient computation on event batches, and we plan to further build on Arrow as standardized data plane—both internally and as high-bandwidth format externally.

Configuration Improvements for Flexible Deployments

The new option vast.start.commands allows for specifying an ordered list of VAST commands that run after successful startup. The effect is the same as (1) first starting a node, and (2) then using a separate VAST client to issue commands. This is useful for commands that have side effects that cannot be expressed through the config file, e.g., starting a source inside the VAST server that listens on a socket or reads packets from a network interface.

Consider the following example:

# <configdir>/vast/vast.yaml
vast:
  plugins:
    - pcap 
  start:
    commands:
      - spawn source pcap
  import:
    pcap:
      interface: en0

Previously when deploying VAST, operators needed to first start the VAST server process, wait for it to be ready, and then issue a vast spawn source pcap command to listen on for PCAPs from inside the server process. The new option streamlines the configuration process.

Static plugins must now be enabled explicitly. Previously, VAST only required enabling dynamic plugins explicitly. The behavior is now unified and allows for static binary deployments with plugins selectively enabled as desired. The reserved plugin names bundled and all load all plugins built alongside VAST (static or dynamic), or all plugins from the configured plugin directories, respectively.

Plugins load their respective configuration from <configdir>/vast/plugin/<plugin-name>.yaml in addition to the regular configuration file at <configdir>/vast/vast.yaml. The new plugin-specific file does not require putting configuration under the key plugins.<plugin-name>. The following two configurations are equivalent, making it possible to install and uninstall plugins alongside their configuration.

# <configdir>/vast/vast.yaml
vast:
  plugins:
    - example

plugins:
  example:
    some-option: value
# <configdir>/vast/vast.yaml
vast:
  plugins:
    - example

# <configdir>/vast/plugins/example.yaml
some-option: value

Lastly, VAST now merges lists from configuration files. E.g., running VAST with --plugins=some-plugin and vast.plugins: [other-plugin] in the configuration now results in both some-plugin and other-plugin instead of just some-plugin.

Smaller Things

  • Import processes no longer hang on receiving SIGINT or SIGKILL.
  • The import csv command handles quoted fields correctly.
  • The vast.log-rotation-threshold option now works as expected.
  • The UDS metrics sink continues to send data when the receiving socket is recreated
  • VAST no longer crashes when querying for string fields with non-string values.

Changelog Highlights

As always, you can find the full scoop in our changelog.

⚡ Breaking Changes

  • Apache Arrow is now a required dependency. The previously deprecated build option -DVAST_ENABLE_ARROW=OFF no longer exists. #1683

  • VAST no longer loads static plugins by default. Generally, VAST now treats static plugins and bundled dynamic plugins equally, allowing users to enable or disable static plugins as needed for their deployments. #1703

⚠️ Changes

  • The tenzir/vast Docker image bundles the PCAP plugin. #1705

  • VAST merges lists from configuration files. E.g., running VAST with --plugins=some-plugin and vast.plugins: [other-plugin] in the configuration now results in both some-plugin and other-plugin being loaded (sorted by the usual precedence), instead of just some-plugin. #1721 #1734

🎁 Features

  • The new option vast.start.commands allows for specifying an ordered list of VAST commands that run after successful startup. The effect is the same as first starting a node, and then using another VAST client to issue commands. This is useful for commands that have side effects that cannot be expressed through the config file, e.g., starting a source inside the VAST server that listens on a socket or reads packets from a network interface. #1699

  • The options vast.plugins and vast.plugin-dirs may now be specified on the command line as well as the configuration. Use the options --plugins and --plugin-dirs respectively. #1703

  • Add the reserved plugin name bundled to vast.plugins to enable load all bundled plugins, i.e., static or dynamic plugins built alongside VAST, or use --plugins=bundled on the command line. The reserved plugin name all causes all bundled and external plugins to be loaded, i.e., all shared libraries matching libvast-plugin-* from the configured vast.plugin-dirs. #1703

  • It's now possible to configure the VAST endpoint as an environment variable by setting VAST_ENDPOINT. This has higher precedence than setting vast.endpoint in configuration files, but lower precedence than passing --endpoint= on the command-line. #1714

  • Plugins load their respective configuration from <configdir>/vast/plugin/<plugin-name>.yaml in addition to the regular configuration file at <configdir>/vast/vast.yaml. The new plugin-specific file does not require putting configuration under the key plugins.<plugin-name>. This allows for deploying plugins without needing to touch the <configdir>/vast/vast.yaml configuration file. #1724

🪲 Bug Fixes

  • VAST no longer crashes when querying for string fields with non-string values. Instead, an error message warns the user about an invalid query. #1685

  • The UDS metrics sink continues to send data when the receiving socket is recreated. #1702

  • Additional tags for the tenzir/vast Docker image for the release versions exist, e.g., tenzir/vast:2021.05.27. #1711

  • The import csv command handles quoted fields correctly. Previously, the quotes were part of the parsed value, and field separators in quoted strings caused the parser to fail. #1712

  • Import processes no longer hang on receiving SIGINT or SIGKILL. Instead, they shut down properly after flushing yet to be processed data. #1718