Skip to content

VAST 2020.06.25
Compare
Choose a tag to compare
@dominiklohmann dominiklohmann released this 25 Jun 09:24
2020.06.25
9e78297
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

We're happy to announce the monthly release 2020.06.25 of VAST. This month we wound up with a good balance between improving robustness and adding new features. Please see the CHANGELOG for a complete list of changes.

Aging Data

The aging feature is now open-source. Aging is the periodic removal of existing data. This helps in situations when you have a disk budget or when there exist data retention policies. We marked this feature as experimental because the deletion currently affects the archive only. Even though the data is no longer materializable that way, the corresponding index entries still exist. Since an index is lossy, cleaning out the data structures in there is actually not trivial. But before marking this feature as stable, we will come up with a solution.

Faster IP Address Queries

We also added an optimization to improve the query latency for IP address point queries. When you query for a specific address, say 6.6.6.6, you now get an instant answer when there is no reference to that IP address in the database. This helps when having multiple queries of the form “did this thing hit us in the past 12 month?” If the address exists, VAST is now much smarter in selecting the relevant index partitions. Internally, we achieved this by adding a new Bloom filter synopsis to the index.

🎁 Features

  • The meta index now uses Bloom filters for equality queries involving IP addresses. This especially accelerates queries where the user wants to know whether a certain IP address exists in the entire database. #931

  • The import command gained a new --read-timeout option that forces data to be forwarded to the importer regardless of the internal batching parameters and table slices being unfinished. This allows for reducing the latency between the import command and the node. The default timeout is 10 seconds. #916

  • VAST now has options to limit the amount of results produced by an invocation of vast explore. #882

  • The import json command's type restrictions are more relaxed now, and can additionally convert from JSON strings to VAST internal data types. #891

  • VAST now supports /etc/vast/vast.conf as an additional fallback for the configuration file. The following file locations are looked at in order: Path specified on the command line via --config=path/to/vast.conf, vast.conf in current working directory, ${INSTALL_PREFIX}/etc/vast/vast.conf, and /etc/vast/vast.conf. #898

🧬 Experimental Features

  • VAST now supports aging out existing data. This feature currently only concerns data in the archive. The options system.aging-frequency and system.aging-query configure a query that runs on a regular schedule to determine which events to delete. It is also possible to trigger an aging cycle manually. #929

⚠️ Changes

  • The options system.table-slice-type and system.table-slice-size have been removed, as they duplicated import.table-slice-type and import.table-slice-size respectively. #908 #951

  • The default table slice type has been renamed to caf. It has not been the default when built with Apache Arrow support for a while now, and the new name more accurately reflects what it is doing. #948

  • The JSON export format now renders timestamps using strings instead of numbers in order to avoid possible loss of precision. #909

🐞 Bug Fixes

  • A bogus import process that assembled table slices with a greater number of events than expected by the node was able to lead to wrong query results. #908

  • A use after free bug would sometimes crash the node while it was shutting down. #896

  • The export json command now correctly unescapes its output. #910

  • VAST now correctly checks for control characters in inputs. #910

Assets 5