Skip to content

VAST 2021.07.29
Compare
Choose a tag to compare
@lava lava released this 30 Jul 07:56
2021.07.29
61cfd32
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

This release of VAST brings a number of exciting experimental features that can be checked out by enterprising users.

Broker Plugin

The new Broker plugin enables seamless log ingestion from Zeek to VAST via a TCP socket. Broker is Zeek's messaging library and the plugin turns VAST into a Zeek logger node. This plugin is bundled by default with our static binaries. Use vast import broker to establish a connection to Zeek node and acquire logs. On the Zeek side, you only need to activate Broker. The nice thing is that you can now also disable file-logging in Zeek:

@load base/frameworks/broker

# Disable logging to local files.
redef Log::enable_local_logging = F;

# Open a listening socket at startup, if you aren’t doing it already.
event zeek_init()
  {
  Broker::listen(Broker::default_listen_address, Broker::default_port,
                 Broker::default_listen_retry);
  }

Store Plugin

With the new store backends, we take our first step towards distributed data stores, allowing users to implement custom store backends that hold the data indexed by a partition. They can be configured using the new vast.store-backend config option in your vast.yaml configuration:

vast:
  store-backend: segment-store

There’s now a clean separation of indexes and raw data, which gives rise to variety of storage lakes, such as S3, Parquet, HBase, Cassandra.

Import Filters

VAST now supports import filter expressions. They act as the dual to export query expressions:

vast import suricata '#type == "suricata.alert"' < eve.json

This invocation will import only suricata.alert events, discarding all other events.

Import filters come in especially handy when processing binary data, such as NetFlow or PCAP, where ASCII-based preprocessing with UNIX tools like jq do not work well. It also gives you “one ring to rule them all”, without the need to pick a tool-specific mechanism to winnow down the ingest feed.

Fixlets & Tweaks

As usual a number of bug fixes and quality-of-life improvements ship with this release, making upgrading highly recommended even for users who are not living on the bleeding edge. A small selection:

  • The [*** LOG ERROR #0001 ***] error message on startup under Linux no longer exists. #1754

  • Import processes now respond quicker. Requests to shutdown when the server process exits are no longer delayed for busy imports, and metrics and telemetry reports are now written in a timely manner. #1771

  • VAST does not abort JSON imports anymore when encountering something other than a JSON object, e.g., a number or a string. Instead, VAST skips the offending line. #1759

  • Queries against fields using a #index=hash attribute will no longer miss some results, due to fixing a bug in the offset computation in bitmap processing. #1755

Last but not least, we had a big round of upgrades on the developer side. For our dependencies, VAST no longer officially supports Debian Buster with GCC-8 and is instead now being tested on Debian Bullseye with GCC-10. The provided Docker images now use debian:bullseye-slim as base image. Users that require Debian Buster support should use the provided static Nix builds instead.

As a side effect, we were also able to upgrade the VAST code base to C++20. At this place we want to include a big shoutout to @JoeLoser, who immediately contributed many PRs to simplify our code base by introducing concepts!

Changelog Highlights

As always, you can find the full scoop in our changelog.

  • Plugin versions are now unique to facilitate debugging. They consist of three optional parts: The CMake project version of the plugin, the Git revision of the last commit that touched the plugin, and a dirty suffix for uncommitted changes to the plugin. Plugin developers no longer need to specify the version manually in the plugin entrypoint.
    #1764

  • VAST now exports per-layout import metrics under <reader>.events.<layout-name> in addition to the regular <reader>.events, making it easier to understand the event type distribution.
    #1781

  • VAST now supports the arm64 architecture.
    #1773

Assets 6
Loading