Prepare the repository for VAST v3.0.1 (#3022)

This prepares the repository for VAST v3.0.1.

changelog/v3.0.1/bug-fixes/3020.md

@@ -0,0 +1,3 @@
+Automatic partition rebuilding both updates partitions with an outdated storage
+format and merges undersized partitions continuously in the background. This now
+also works as expected for outdated but not undersized partitions.

python/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pyvast"
-version = "3.0.0"
+version = "3.0.1"
 description = "A security telemetry engine for detection and response"
 authors = ["Tenzir <engineering@tenzir.com>"]
 maintainers = ["Tenzir <engineering@tenzir.com>"]

scripts/prepare-release

@@ -25,15 +25,18 @@ if ! [ -z "$(git status --porcelain)" ]; then
   exit 1
 fi
 
-# Determine source and build directory.
+# Determine the source directory.
 source_dir="$(git -C "$(dirname "$0:A")" rev-parse --show-toplevel)"
-build_dir=$(mktemp -d 2>/dev/null || mktemp -d -t "build-${new_version}")
 
 # Determine the last version including release candidates, the last version
 # excluding release candidates and the new version.
 last_rc_version="$(git -C "${source_dir}" describe --abbrev=0 --match='v[0-9]*')"
 last_version="$(git -C "${source_dir}" describe --abbrev=0 --match='v[0-9]*' --exclude='*-rc*')"
 new_version="$1"
+new_version_major_minor="${new_version%.*}"
+
+# Determine the build directory.
+build_dir=$(mktemp -d 2>/dev/null || mktemp -d -t "build-${new_version}")
 
 # Check that the new version does not exist already.
 if [ -d "${source_dir}/changelog/${new_version}" ]; then
@@ -58,7 +61,7 @@ if [[ "${new_version}" != *"-rc"* ]]; then
   perl -i -pe "s/${last_version}/${new_version}/g" "${source_dir}/README.md"
   pushd "${source_dir}/web"
   yarn install --frozen-lockfile
-  yarn docusaurus docs:version "${new_version}"
+  yarn docusaurus docs:version "${new_version_major_minor}"
   popd
 fi
 

version.json

@@ -4,14 +4,14 @@
     "annotated git tag without the leading 'v'.",
     "This value gets updated automatically by `scripts/prepare-release`."
   ],
-  "vast-version-fallback": "3.0.0",
+  "vast-version-fallback": "3.0.1",
   "vast-version-rev-count_COMMENT": [
     "This value stores the ancestor count of the tagged commit, calculated",
     "with `git rev-list --count HEAD`, then incremented by 1. This operates",
     "under the assumption that the release-preparing PR contains exactly one",
     "commit and is rebased before merging."
   ],
-  "vast-version-rev-count": 15630,
+  "vast-version-rev-count": 15723,
   "vast-partition-version_COMMENT": [
     "The partition version. This number must be bumped alongside the release",
     "version for releases that contain major format changes to the on-disk",

web/blog/vast-v3.0/index.md

@@ -12,7 +12,7 @@ to the the VAST language, making it easy to write down dataflow pipelines that
 filter, reshape, aggregate, and enrich security event data. Think of VAST as
 security data pipelines plus open storage engine.
 
-[github-vast-release]: https://github.com/tenzir/vast/releases/tag/v3.0.0
+[github-vast-release]: https://github.com/tenzir/vast/releases/tag/v3.0.1
 
 <!--truncate-->
 
@@ -325,7 +325,7 @@ user-facing changes since the last release.
 
 Here are some entries that we want to highlight:
 
-[changelog]: //vast.io/changelog#v300
+[changelog]: https://vast.io/changelog#v301
 
 ### Removing Empty Fields from JSON Output
 

web/versioned_docs/version-v3.0.0/contribute/changelog.md → web/versioned_docs/version-v3.0/contribute/changelog.md

@@ -10,7 +10,7 @@ noteworthy *user-facing* changes.
 The procedure for adding a changelog entry looks as follows.
 
 1. Open your pull request with your proposed changes
-2. Go to the [`changelog`](https://github.com/tenzir/vast/tree/master/changelog)
+2. Go to the [`changelog`](https://github.com/tenzir/vast/tree/main/changelog)
    directory in the top-level repository directory and navigate to the
    `next` sub-directory.
 3. Choose a category for your changes and go to the corresponding sub-directory:

web/versioned_docs/version-v3.0.0/contribute/documentation.md → web/versioned_docs/version-v3.0/contribute/documentation.md

@@ -6,13 +6,13 @@ sidebar_position: 5
 
 The VAST documentation resides inside [our main GitHub
 repository](https://github.com/tenzir/vast) in
-[`/web/docs`](https://github.com/tenzir/vast/tree/master/web/docs).
+[`/web/docs`](https://github.com/tenzir/vast/tree/main/web/docs).
 We use [Docusaurus](https://docusaurus.io/) as website framework.
 
 ## Build and view locally
 
 To view the entire site (including the documentation) locally,
-change to the [`/web`](https://github.com/tenzir/vast/tree/master/web/)
+change to the [`/web`](https://github.com/tenzir/vast/tree/main/web/)
 directory and invoke [`yarn`](https://yarnpkg.com/), or to be on the safe side,
 `yarn install --frozen-lockfile` to avoid pollution from global dependencies.
 Then build and serve the site via:

web/versioned_docs/version-v3.0.0/contribute/notebooks.md → web/versioned_docs/version-v3.0/contribute/notebooks.md

@@ -43,7 +43,7 @@ features within a Quarto notebook.
 
 Other services can be added to the context of the Quarto notebook execution by
 extending the Docker Compose setup with [extra
-overlays](https://github.com/tenzir/vast/tree/master/docker/).
+overlays](https://github.com/tenzir/vast/tree/main/docker/).
 
 The website build harness uses this Docker Compose environment to run Quarto
 notebooks that represent more elaborate user guides or blog posts that. For

web/versioned_docs/version-v3.0.0/develop/guides/write-a-plugin.md → web/versioned_docs/version-v3.0/develop/guides/write-a-plugin.md

@@ -23,9 +23,9 @@ builds.
 
 VAST ships with an example plugin that showcases how a typical scaffold looks
 like. Have a look at the the [example
-plugins](https://github.com/tenzir/vast/tree/master/examples/plugins) directory,
+plugins](https://github.com/tenzir/vast/tree/main/examples/plugins) directory,
 and an [example `CMakeLists.txt` file for
-plugins](https://github.com/tenzir/vast/blob/master/examples/plugins/analyzer/CMakeLists.txt).
+plugins](https://github.com/tenzir/vast/blob/main/examples/plugins/analyzer/CMakeLists.txt).
 
 We highly urge calling the provided `VASTRegisterPlugin` CMake in your plugin's
 `CMakeLists.txt` file instead of handrolling your CMake build scaffolding
@@ -61,7 +61,7 @@ diamond](https://isocpp.org/wiki/faq/multiple-inheritance#mi-diamond).
 :::
 
 Please also consult the [example-analyzer
-plugin](https://github.com/tenzir/vast/tree/master/examples/plugins/analyzer)
+plugin](https://github.com/tenzir/vast/tree/main/examples/plugins/analyzer)
 for a complete end-to-end code example.
 
 ## Implement the plugin interface

web/versioned_docs/version-v3.0.0/setup/deploy/ansible.md → web/versioned_docs/version-v3.0/setup/deploy/ansible.md

@@ -12,7 +12,7 @@ The role definition is in the [`ansible/roles/vast`][vast-repo-ansible]
 directory of the VAST repository. You need a local copy of this directory so you
 can use it in your playbook.
 
-[vast-repo-ansible]: https://github.com/tenzir/vast/tree/master/ansible/roles/vast
+[vast-repo-ansible]: https://github.com/tenzir/vast/tree/main/ansible/roles/vast
 
 ## Example
 

web/versioned_docs/version-v3.0.0/setup/deploy/aws.md → web/versioned_docs/version-v3.0/setup/deploy/aws.md

@@ -26,7 +26,7 @@ illustrates the high-level architecture:
 :::info Source Code Required
 Make sure you have [downloaded the VAST source code](../download.md)
 and change into the directory
-[cloud/aws](https://github.com/tenzir/vast/tree/master/cloud/aws) that contains
+[cloud/aws](https://github.com/tenzir/vast/tree/main/cloud/aws) that contains
 all deployment scripts.
 :::
 
@@ -98,9 +98,9 @@ To tear everything down, use:
 ./vast-cloud destroy
 ```
 
-[vast-cloud-dockerfile]: https://github.com/tenzir/vast/blob/master/cloud/aws/docker/cli.Dockerfile
-[vast-cloud-script]: https://github.com/tenzir/vast/blob/master/cloud/aws/vast-cloud
-[core.py]: https://github.com/tenzir/vast/blob/master/cloud/aws/cli/core.py
+[vast-cloud-dockerfile]: https://github.com/tenzir/vast/blob/main/cloud/aws/docker/cli.Dockerfile
+[vast-cloud-script]: https://github.com/tenzir/vast/blob/main/cloud/aws/vast-cloud
+[core.py]: https://github.com/tenzir/vast/blob/main/cloud/aws/cli/core.py
 
 :::warning Caveats
 - Access to the VAST server is enforced by limiting inbound traffic to its local

web/versioned_docs/version-v3.0.0/setup/deploy/docker-compose.md → web/versioned_docs/version-v3.0/setup/deploy/docker-compose.md

@@ -8,7 +8,7 @@ We offer a range of Docker Compose files for quickly getting up and running with
 VAST. All mentioned files are in the [`docker`][vast-repo-docker] directory of
 the VAST repository, and require having the repository checked out locally.
 
-[vast-repo-docker]: https://github.com/tenzir/vast/tree/master/docker
+[vast-repo-docker]: https://github.com/tenzir/vast/tree/main/docker
 
 :::info Docker Compose V2 CLI
 All examples shown use the [Docker Compose V2 CLI][docker-compose-v2-cli]. If

web/versioned_docs/version-v3.0.0/setup/deploy/docker.md → web/versioned_docs/version-v3.0/setup/deploy/docker.md

@@ -77,7 +77,7 @@ docker run -e VAST_ENDPOINT -e VAST_IMPORT__BATCH_SIZE=42 --env-file .env \
 You can always build your own Docker image in case our prebuilt images don't fit
 your use case.
 
-Our official [Dockerfile](https://github.com/tenzir/vast/blob/master/Dockerfile)
+Our official [Dockerfile](https://github.com/tenzir/vast/blob/main/Dockerfile)
 offers two starting points: a *development* and *production* layer.
 
 Before building the image, make sure to fetch all submodules:

web/versioned_docs/version-v3.0.0/setup/download.md → web/versioned_docs/version-v3.0/setup/download.md

@@ -26,7 +26,7 @@ release and the current development version.
 </div>
 
 We also offer prebuilt statically linked binaries for every Git commit to the
-`master` branch.
+`main` branch.
 
 ```bash
 version="$(git describe --abbrev=10 --long --dirty --match='v[0-9]*')"

web/versioned_docs/version-v3.0.0/understand/language/pipelines.md → web/versioned_docs/version-v3.0/understand/language/pipelines.md

@@ -88,23 +88,20 @@ example:
 Here is how you write this pipeline in the VAST language:
 
 ```cpp
+/* 1. Get data from VAST */
 from vast
+/* 2. Filter out a subset of events */
 | where #type == "zeek.weird" && note == "SSL::Invalid_Server_Cert"
+/* 3. Aggregate them by destination IP */
 | summarize count(num) by id.resp_h
+/* 4. Sort by frequency */
 | sort
+/* 5. Take the top-20 items */
 | head 20
+/* 6. Write the output as JSON to standard output */
 | write json to stdout
 ```
 
-The pipeline performs the following steps:
-
-1. Get data from VAST
-2. Filters out a subset of events
-3. Aggregate them by destination IP
-4. Sort by frequency
-5. Take the top-20 items
-6. Write the output as JSON to standard output
-
 :::caution Running closed pipelines
 We do not support running closed pipelines yet, but aim to ship this ability
 soon. The corresponding [roadmap

web/versions.json

@@ -1,3 +1,3 @@
 [
-  "v3.0.0"
+  "v3.0"
 ]