CHANGELOG.md

Changelog

This changelog documents all notable user-facing changes of pyvast-threatbus.

Every entry has a category for which we use the following visual abbreviations:

• 🎁 Features
• 🧬 Experimental Features
• ⚠️ Changes
• ⚡️ Breaking Changes
• 🐞 Bug Fixes

[2022.05.16]

• ⚠️ To make use of VAST's new query query scheduler vast-threatbus now runs up to 500 queries in parallel by default. It also waits for one hour before aborting a query. #190

2022.01.27

No user-facing changes.

2021.12.16

• 🎁 Added a new field %matchtype to the transform_context setting of vast-threatbus that can be used to distinguish between live and retro matches.

• ⚠️ vast-threatbus no longer adds a source field to the x_threatbus_sighting_context field of generated sightings.

• ⚠️ vast-threatbus now adds the matched event to the x_threatbus_sighting_context for results coming from live matcher.

• 🐞 vast-threatbus now handles sightings generated by VAST matchers using probabilistic filters. Note that to correlate sighting and indicator an exact filter must be used, as probabilistic filters do not store the STIX identifier of the indicator. VAST Threat Bus emits the invalid identifier note--00000000-0000-4000-8000-000000000000 for sightings generated from matchers with probabilistic filters. #176 #178

2021.11.22

• 🐞 vast-threatbus no longer fails to check for the availability of low priority queries. #173

2021.11.18

• ⚠️ vast-threatbus now issues low priority queries if those are supported by the version of VAST. #171

2021.09.30

No user-facing changes.

2021.08.26

• ⚡️ We renamed PyVAST Threat Bus to VAST Threat Bus for clarity. The PyPI package name and the binary name change from pyvast-threatbus to vast-threatbus accordingly. #159

• 🎁 Live matching with VAST works again! #156

• ⚠️ vast-threatbus now depends on version 3.0 of stix2 package. #162

2021.07.29

• ⚠️ The metric for indicator query time now only reflects the actual time spent querying VAST and does not regard unstarted VAST queries any longer. #145

• ⚠️ Metrics sent by pyvast-threatbus used the short hostname as given by socket.gethostname(). This has been changed to use socket.getfqdn(). #144

• ⚠️ The Dockerfile of pyvast-threatbus has moved to the repository toplevel and now installs Threat Bus from source. This way, the Docker build always uses the latest Threat Bus sources, instead of the latest version from PyPI. #141

• 🐞 pyvast-threatbus now implements the new zmq management protocol of the threatbus-zmq-app plugin. The app now simply re-uses the Threat Bus hostname as it is configured in the users config.yaml and appends the port specifications for pub and sub communication that it receives as part of the subscription success response. #140

• 🐞 The metrics value serialization in pyvast-threatbus contained spaces in the fields of the measurements, which is not valid according to the line protocol spec and caused the measurement to be rejected. We fixed the format to ensure all fields are separated by commas. #136

• ⚠️ pyvast-threatbus now uses Dynaconf for configuration management. Configuration via a config file works exactly as it has worked before. Users can provide a path to the config file using the -c option. pyvast-threatbus now considers files named config.yaml and config.yml as default configs if located in the same directory. Additionally, pyvast-threatbus now supports configration via environment variables and .dotenv. Env vars need to be prefixed with PYVAST_THREATBUS_ to be respected and always take precedence over values in config files. #133

2021.06.24

• ⚡️ pyvast-threatbus now uses point queries over substring queries for URI indicators, because such queries are much faster. This may result in less matches than before. E.g., a URI indicator tenzir.com that used to match docs.tenzir.com as well as https://tenzir.com now only matches exactly the indicator. #130

• 🎁 pyvast-threatbus now collects metrics about received indicators that are about to be matched retrospectively against VAST. The new metric is called retro_match_backlog and allows users to determine if a backlog is building up. A backlog builds when pyvast-threatbus hits the user-configured limit of max backgroud tasks while at the same time VAST responds slowly to the issued queries. #129

• 🎁 pyvast-threatbus now comes with its own Dockerfile. Pre-built images are available on Dockerhub. #126

2021.05.27

• ⚠️ All Threat Bus apps that connect via ZeroMQ like pyvast-threatbus. now shutdown gracefully and do not longer print a stack trace when receiving any stop signal. #118

• ⚠️ The -c / --config parameter is now explicitly required to start pyvast-threatbus. Starting without it will print a helpful error message. #119

• ⚠️ pyvast-threatbus now uses the timestamp of retro- & live-matches to set the last_seen property of STIX-2 Sightings, instead of setting the created timestamp. The created timestamp now always refers to the actual creation time of the sightings. #117

2021.04.29

• 🐞 We fixed an unhandled exception in the post-processing of sighting context data for both retro- and live-matched sightings. The bug was introduced with the STIX-2 rewrite and effectively rendered both the transform_context and sink options unusable. #112

• 🎁 pyvast-threatbus now supports a new config option to set timeouts for VAST retro-queries: retro_match_timeout. Pending queries are killed upon timeout. VAST results that were exported before the timeout hit are still reported as valid Sightings. #110

• ⚠️ VAST's proprietary Threat Intel Matching feature was rewritten as a VAST plugin. pyvast-threatbus now works with the changed command line interface. #109

• 🐞 We fixed a bug where VAST matcher results where passed to the wrong mapping function after being retrieved from VAST. #109

2021.03.25

• 🎁 pyvast-threatbus now supports the STIX-2 (version 2.1) standard for Indicators and Sightings. The app converts STIX-2 Indicators on best-effort basis to both VAST queries and VAST matcher IoCs to support both retro- and live-matching. Likewise, pyvast-threatbus converts VAST query results as well as VAST matcher sightings to valid STIX-2 Sightings before publishing them on Threat Bus topics. #105

• ⚠️ The retro-matching now applies a strict equality comparison when mapping IoCs to VAST queries. Prior to this change pyvast-threatbus used substring search, which came at heavy runtime costs when issuing hundreds of queries per second. #104

2021.02.24

• 🐞 Users can now run retro-queries with an unbounded number of results against VAST by setting the retro_match_max_events parameter to 0. #98

• ⚠️ Users now can use both, retro-matching and live-matching with VAST simultaneously for any given IoC. On the flip side, there is no longer a default mode of operation. To use live-matching, users now must specifically configure it via setting live_match: true in their config.yaml file. #95

• ⚠️ pyvast-threatbus drops support to unflatten JSON that it receives from vast export because VAST can now return unflattened JSON by default. #92

2020.12.16

• 🎁 pyvast-threatbus now supports basic metric collection. It stores metrics in influx line protocol syntax in a configurable file on disk. #85

• ⚠️ pyvast-threatbus cannot be started with command line arguments anymore. From now on, the application only supports one option, -c, to pass a config file. #85

• 🎁 pyvast-threatbus now uses the Threat Bus logger module. Users can configure logging the same way as in Threat Bus, via a logging section in the config.yaml file. #80

2020.11.26

• 🐞 pyvast-threatbus escapes backslashes and quotes in IoCs before it queries VAST. #74

• 🎁 pyvast-threatbus now uses asynchronous background tasks to query VAST concurrently. VAST queries were executed sequentially prior to this change. This boosts the performance by the factor of allowed concurrent background tasks. Users can control the maximum number of concurrent background tasks with the new max-background-tasks configuration option. #61

• 🎁 The Python app to connect VAST with Threat Bus is now packaged and published on PyPI. You can install the package via pip install pyvast-threatbus. #63