Skip to content

VAST 2020.05.28
Compare
Choose a tag to compare
@dominiklohmann dominiklohmann released this 28 May 13:52
2020.05.28
2fc7585
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

We’re happy to announce our release 2020.05.28 this month. We are making progress towards our goals of automated telemetry exploration and have begun with experimental work on temporal and spatial contextualization in the backend as well as our frontend.

This month brings two new features to VAST: An early version of the explore command, and an improvement of type deduction during JSON import.

As always, see the CHANGELOG for a full list of changes.

Exploring time and space with VAST

We're working on a new vast command called explore. This command provides additional context for the results of a search query in VAST. For example, you can look at all records within 30 seconds of a connection from a supposedly suspicious host 10.0.0.42 by using the following command:

vast explore --after 30s 'src_ip == 10.0.0.42'

This command is still under active development, so changes to the user interface as well as performance improvements are expected in the upcoming versions. Have a look at the documentation page for more information.

Importing heterogenous JSONL with automatic type deduction

Before this release, users had to explicitly specify the VAST even type during JSON import:

# import JSON for the zeek.conn type
vast import json -t zeek.conn -r path/to/conn.log.json

Aside from being inconvenient to use, this made it impossible to import heterogeneous line-delimited JSON data with different event types. We changed the behavior of -t in that it now represents a prefix of types to consider for type deduction. The deduction itself happens by inspecting the available field names. When -t only yields a single type, VAST keeps the old behavior of filling in nil values for unmatched fields. This means that specifying -t at all is now mostly unnecessary for importing data with a known schema.

🎁 Features

  • When importing JSON data without knowing the type of the imported events a priori, VAST now supports automatic event type deduction based on the JSON object keys in the data. VAST selects a type iff the set of fields match a known type. The --type / -t option to the vast import command restricts the matching to the set of types that share the provided prefix. Omitting -t attempts to match JSON against all known types. If only a single variant of a type is matched, the import falls back to the old behavior and fills in nil for mismatched keys. #875
  • When importing events of a new or updated type, VAST now only requires the type to be specified once (e.g., in a schema file). For consecutive imports, the event type does not need to be specified again. A list of registered types can now be viewed using vast status under the key node.type-registry.types. #875
  • All input parsers now support mixed \n and \r\n line endings. #865
  • VAST now ships with a schema suitable for Sysmon import. #886

🧬 Experimental Features

  • Added a new 'explore' command to VAST that can be used to show data records within a certain time from the results of a query. #873 #877

⚠️ Changes

  • Spreading a query over multiple command line arguments in commands like explore/export/pivot/etc. has been deprecated. #878
  • The command line flag for disabling the accountant has been renamed to --disable-metrics to more accurately reflect its intended purpose. The internal vast.statistics event has been renamed to vast.metrics. #870

🐞 Bug Fixes

  • Fixed a bug that caused vast import processes to produce default table slices, despite having the arrow type as the default. #866
  • Fixed a bug where setting the logger.file-verbosity in the config file would not have an effect. #866

🐞 Bug Fixes (Pro only)

  • The MsgPack table slice failed to decode enumeration types. Now encoding and decoding works as expected.
Assets 5
Loading