VAST 2021.05.27

We're happy to announce our release 2021.05.27 of VAST.

Apache Arrow

This is the last release of VAST that supports disabling Apache Arrow at build time. As the Arrow ecosystem matures, we feel confident in making it a required dependency and plan to leverage extensively in the future.

Transforms

VAST now supports import and export transforms. Using transforms allows for applying automatic data transformations as events enter or leave VAST, such as removing, hashing, or encrypting specific fields. All transformations can be configured to run at either the client or the server side. These are the possible combinations.

Location	Transform Type	Use Case	Example
Client	Import	Enrichment	Add Community ID to flow telemetry
Server	Import	Compliance	Anonymize PII data
Client	Export	Post-processing	Compute expensive function (e.g., string entropy)
Server	Export	Access Control	Remove sensitive fields

Our user-facing documentation on transforms explains how to use the feature.

A transform is a named, composable list of steps that execute sequentially. VAST ships with a new plugin type to write custom transformations in C++. Learn more about developing transform plugins at our developer-facing documentation.

Smaller Things

The new --timeout option for the export command allows for setting a timeout for VAST queries. If a query still executes within the provided time frame, the client will cancel the execution. VAST now correctly runs on Linux distributions where the library install directory is lib64 instead of lib. The status command now prints the VAST server version information. VAST now issues a warning when the local and remote versions differ.

Changelog Highlights

As always, you can find the full scoop in our changelog.

⚡ Breaking Changes

• Schemas are no longer implicitly shared between sources, i.e., an import process importing data with a custom schema will no longer affect other sources started at a later point in time. Schemas known to the VAST server process are still available to all import processes. We do not expect this change to have a real-world impact, but it could break setups where some sources have been installed on hosts without their own schema files, the VAST server did not have up-to-date schema files, and other sources were (ab)used to provide the latest type information. #1656

🎁 Features

• The new transforms feature allows VAST to apply transformations to incoming and outgoing data. A transform consists of a sequence of steps that execute sequentially, e.g., to remove, overwrite, hash, encrypt data. A new plugin type makes it easy to write custom transforms. #1517 #1656

• Plugin schemas are now installed to <datadir>/vast/plugin/<plugin>/schema, while VAST's built-in schemas reside in <datadir>/vast/schema. The load order guarantees that plugins are able to reliably override the schemas bundled with VAST. #1608

• The new option vast export --timeout=<duration> allows for setting a timeout for VAST queries. Cancelled exports result in a non-zero exit code. #1611

• To enable easier post-processing, the new option vast.export.json.numeric-durations switches JSON output of duration types from human-readable strings (e.g., "4.2m") to numeric (e.g., 252.15) in fractional seconds. #1628

• The status command now prints the VAST server version information under the version key. #1652

• The new setting vast.disk-monitor-step-size enables the disk monitor to remove N partitions at once before re-checking if the new size of the database directory is now small enough. This is useful when checking the size of a directory is an expensive operation itself, e.g., on compressed filesystems. #1655

🪲 Bug Fixes

• VAST now correctly refuses to run when loaded plugins fail their initialization, i.e., are in a state that cannot be reasoned about. #1618