VAST 2021.12.16

Dear users, we are happy to announce VAST 2021.12.16! Since the past release we’ve heavily focused on performance and correctness, and have achieved a sizable performance improvement of 2x on the ingest and query paths. More below! 👇

Performance Improvements

With this release focussing on performance, we ran the following import/export benchmarks that mimic typical workloads when processing large amounts of telemetry during security investigations:

• Import a large Suricata Eve JSON log (5.5 GB NDJSON containing 8,557,668 events)
• Export DNS events from a given source IP to a destination subnet
• Export all Suricata events whose destination IP is in a given subnet
• Export all Suricata events that originate from HTTPS traffic

We’ve started with a clean database for both the 2021.11.18 and the 2021.12.16 versions of VAST. We ran each query in isolation, i.e., during the benchmark run for a particular query the VAST server process was otherwise idle.

The dataset contained the following event distribution:

• suricata.dns: 671,418 (7.85%)
• suricata.fileinfo: 391,104 (4.57%)
• suricata.flow: 6,909,807 (80.74%)
• suricata.ftp: 35,025 (0.41%)
• suricata.http: 457,826 (5.35%)
• suricata.smtp = 3,713 (0.04%)
• suricata.stats = 32 (0.00%)
• suricata.tls = 88,743 (1.04%)

We ran the following benchmark queries:

• Data import: vast import -r <json file> suricata
• DNS traffic query – 2,955 events selected (0.035%): vast export null "net.src.ip == 10.0.0.7 && net.dst.ip !in 10.0.0.0/24 && #type == \"suricata.dns\""
• All outgoing traffic from a /24 network – 10,000 results (via -n 10000) from 5M matches (0.12%): vast export -n 10000 null 'net.src.ip in 10.0.0.0/24 && net.dst.ip !in 10.0.0.0/24'
• All connections to a host in a specific subnet – 21,631 events selected (0.25%): vast export null 'net.dst.ip in 148.0.0.0/24'
• All TCP traffic to port 443 – 447,590 events selected (5.3%): vast export null 'proto == "TCP" && net.dst.port == 443'

Benchmark	2021-12-16	2021-11-18	difference
Data Import	121.475s ± 1.055s	235.210s ± 0.834s	-48%
DNS traffic query	982.8ms ± 11.2ms	1725.5 ± 19.5ms	-43%
All outgoing traffic from a /24 network	1329.3ms ± 730.0ms	17008.4ms ± 390.1ms	-92%
All connections to a host	988.3ms ± 2.2ms	1730.8ms ± 11.6ms	-42%
All TCP traffic to port 443	3009.8ms ± 363.4ms	6240.0ms ± 16.7ms	-52%

In summary, we see a performance improvement by about 2x for both data ingestion and query performance.

We ran the benchmark suite multiple times. The benchmark suite runs every command twice for warm up and then ten times for measurements.

The subnet query has a surprisingly high standard deviation. We initially thought that this is the result of a small sample size, but found the same variance when inflating the dataset by a factor of 10 through repeated ingestion. Our suspicion is that non-determinism in the asynchronous query execution causes side effects due to scattered access patterns. We plan to investigate this deeper in the future.

Type System and Correctness

We’ve revamped VAST’s type system for improved correctness and performance across the board. Generally speaking, whenever a qualified field (e.g., zeek.conn.conn.id denotes the field id in the type of the field conn in the type zeek.conn) is prefix- or suffix-matched, VAST now only matches if the qualified field name is matched exactly or up to a delimiting dot.

This means that vast import --type=zee <format> (sic) will no longer match the zeek.conn type, and neither will #field == “onn.id” (sic) match the conn.id field any longer. We expect this to more closely match the expected behavior for our users.

From a developer perspective, this allowed for an improved internal API that makes it easier to write correct and performant code. We expect the type system changes to have a major positive impact on import performance for some types, and a minor positive impact on export performance and memory utilization. VAST now writes new data in the new improved format, while still supporting the legacy data format. We expect the performance to slightly improve for data access as old data is continuously rotated out by the disk monitor and more of the database is written in the new data format. To the user, this should be completely transparent.

Configurable JSON Selector

The JSON import gained support for user-specified field selectors via the vast import json --selector=<field>[:<prefix>] option. This exposes functionality previously hard-coded for Suricata Eve JSON and Zeek Streaming JSON as the vast import suricata and vast import zeek-json commands (which will continue to be available) in a user-configurable way. The following commands do the exact same thing now:

# Import Suricata Eve JSON, taking the event type from the field 'event_type', and prefixing it with 'suricata.' to find the corresponding VAST type.
vast import json --selector=event_type:suricata
vast import suricata

# Import Zeek Streaming JSON, taking the event type from the field '_path', and prefixing it with 'zeek.' to find the corresponding VAST type.
vast import json --selector=_path:zeek
vast import zeek-json

This will make onboarding of new JSON-based data sources easier. Here’s a full-fledged example with a custom schema:

// A custom DNS event. The ‘event_type’ field always contains the string “dns”.
type custom.dns = record {
  timestamp: timestamp,
  event_type: string,
  src_hostname: string,
  src_ip: addr,
  dns: record {
    rrname: string,
    rrtype: string,
    grouped: record {
      A: string,
    },
  },
}

// A custom URL event. The ‘event_type’ field always contains the string “url”.
type custom.url = record {
  timestamp: timestamp,
  event_type: string,
  src_ip: addr,
  src_port: port,
  dest_ip: addr,
  dest_port: port,
  http: record {
    hostname: string,
    url: string,
  },
}

Here’s some sample data that can be ingested using vast import json --selector=event_type:custom:

{“timestamp”: “2021-12-07T16:21:51.270097+0000”, “event_type”: “dns”, “dns”: {“rrname”: “tenzir.com”}}
{“timestamp”: “2021-12-07T13:14:34.978126+0000”, “event_type”: “url”, “src_ip”: “185.199.109.153”, “http.hostname”: “tenzir.com”}

Smaller Things

• We’ve done some improvements on the VAST Matcher plugin, which now pushes sightings as soon as they’re generated to clients rather than buffering them by an uncontrollable amount of time determined by the streaming layer in CAF. Additionally, we’ve eliminated a memory leak when using the dcso-bloom filter type.
• VAST metrics now contain a version number, and we emit metrics when starting and stopping VAST, which makes it easier to identify deployments on metrics dashboards.

And of course, a whole lot of bug-fixes and stability improvements. Read the full CHANGELOG here.