VAST 2021.06.24
We’re happy to announce our release 2021.06.24 of VAST. This month we focused primarily on maintenance and streamlining of existing functionality.
Dependency Changes
Apache Arrow is now a required VAST dependency. Transforms already leverage Arrow for efficient computation on event batches, and we plan to further build on Arrow as standardized data plane—both internally and as high-bandwidth format externally.
Configuration Improvements for Flexible Deployments
The new option vast.start.commands
allows for specifying an ordered list of VAST commands that run after successful startup. The effect is the same as (1) first starting a node, and (2) then using a separate VAST client to issue commands. This is useful for commands that have side effects that cannot be expressed through the config file, e.g., starting a source inside the VAST server that listens on a socket or reads packets from a network interface.
Consider the following example:
# <configdir>/vast/vast.yaml
vast:
plugins:
- pcap
start:
commands:
- spawn source pcap
import:
pcap:
interface: en0
Previously when deploying VAST, operators needed to first start the VAST server process, wait for it to be ready, and then issue a vast spawn source pcap
command to listen on for PCAPs from inside the server process. The new option streamlines the configuration process.
Static plugins must now be enabled explicitly. Previously, VAST only required enabling dynamic plugins explicitly. The behavior is now unified and allows for static binary deployments with plugins selectively enabled as desired. The reserved plugin names bundled
and all
load all plugins built alongside VAST (static or dynamic), or all plugins from the configured plugin directories, respectively.
Plugins load their respective configuration from <configdir>/vast/plugin/<plugin-name>.yaml
in addition to the regular configuration file at <configdir>/vast/vast.yaml
. The new plugin-specific file does not require putting configuration under the key plugins.<plugin-name>
. The following two configurations are equivalent, making it possible to install and uninstall plugins alongside their configuration.
# <configdir>/vast/vast.yaml
vast:
plugins:
- example
plugins:
example:
some-option: value
# <configdir>/vast/vast.yaml
vast:
plugins:
- example
# <configdir>/vast/plugins/example.yaml
some-option: value
Lastly, VAST now merges lists from configuration files. E.g., running VAST with --plugins=some-plugin
and vast.plugins: [other-plugin]
in the configuration now results in both some-plugin
and other-plugin
instead of just some-plugin
.
Smaller Things
- Import processes no longer hang on receiving SIGINT or SIGKILL.
- The
import csv
command handles quoted fields correctly. - The
vast.log-rotation-threshold
option now works as expected. - The UDS metrics sink continues to send data when the receiving socket is recreated
- VAST no longer crashes when querying for string fields with non-string values.
Changelog Highlights
As always, you can find the full scoop in our changelog.
⚡ Breaking Changes
-
Apache Arrow is now a required dependency. The previously deprecated build option
-DVAST_ENABLE_ARROW=OFF
no longer exists. #1683 -
VAST no longer loads static plugins by default. Generally, VAST now treats static plugins and bundled dynamic plugins equally, allowing users to enable or disable static plugins as needed for their deployments. #1703
⚠️ Changes
-
The tenzir/vast Docker image bundles the PCAP plugin. #1705
-
VAST merges lists from configuration files. E.g., running VAST with
--plugins=some-plugin
andvast.plugins: [other-plugin]
in the configuration now results in bothsome-plugin
andother-plugin
being loaded (sorted by the usual precedence), instead of justsome-plugin
. #1721 #1734
🎁 Features
-
The new option
vast.start.commands
allows for specifying an ordered list of VAST commands that run after successful startup. The effect is the same as first starting a node, and then using another VAST client to issue commands. This is useful for commands that have side effects that cannot be expressed through the config file, e.g., starting a source inside the VAST server that listens on a socket or reads packets from a network interface. #1699 -
The options
vast.plugins
andvast.plugin-dirs
may now be specified on the command line as well as the configuration. Use the options--plugins
and--plugin-dirs
respectively. #1703 -
Add the reserved plugin name
bundled
tovast.plugins
to enable load all bundled plugins, i.e., static or dynamic plugins built alongside VAST, or use--plugins=bundled
on the command line. The reserved plugin nameall
causes all bundled and external plugins to be loaded, i.e., all shared libraries matchinglibvast-plugin-*
from the configuredvast.plugin-dirs
. #1703 -
It's now possible to configure the VAST endpoint as an environment variable by setting
VAST_ENDPOINT
. This has higher precedence than settingvast.endpoint
in configuration files, but lower precedence than passing--endpoint=
on the command-line. #1714 -
Plugins load their respective configuration from
<configdir>/vast/plugin/<plugin-name>.yaml
in addition to the regular configuration file at<configdir>/vast/vast.yaml
. The new plugin-specific file does not require putting configuration under the keyplugins.<plugin-name>
. This allows for deploying plugins without needing to touch the<configdir>/vast/vast.yaml
configuration file. #1724
🪲 Bug Fixes
-
VAST no longer crashes when querying for string fields with non-string values. Instead, an error message warns the user about an invalid query. #1685
-
The UDS metrics sink continues to send data when the receiving socket is recreated. #1702
-
Additional tags for the tenzir/vast Docker image for the release versions exist, e.g.,
tenzir/vast:2021.05.27
. #1711 -
The
import csv
command handles quoted fields correctly. Previously, the quotes were part of the parsed value, and field separators in quoted strings caused the parser to fail. #1712 -
Import processes no longer hang on receiving SIGINT or SIGKILL. Instead, they shut down properly after flushing yet to be processed data. #1718